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TACTICAL  TEAMRESOmOEMANAGEM^  EFFECTS 
ON  COMBATMISSIONIRAINING  PERFORMANCE 

INTRODUCTION 

This  report  describes  the  results  of  research  that  examined  the  relationship  between  team 
coordination  and  tactical  performance  during  a  simulated  mission  that  was  conducted  as  a  natural 
phase  of  Special  Operations  Forces  (SOF)  combat  mission  training  (CMT).  The  work  was 
performed  at  the  58th  Training  Support  Squadron  (58  TRSS),  JCirtland  Air  Force  Base  (KAFB), 
New  Mexico,  using  their  state-of-the-art  simulation  complex  which  is  capable  of  supporting 
CMT  and  mission  rehearsal  (MR). 


Overview 

In  Spiker,  Tourville,  Silverman,  and  Nullmeyer  (1996),  we  presented  a  conceptual 
framework  for  the  measurement  and  evaluation  of  Crew  Resource  Management  (CRM)-based 
skills.  This  framework  expanded  the  scope  of  traditional  CRM  analyses  to  encompass  the  actions 
of  an  entire  combat  team  and  established  the  methodological  foundations  for  the  present 
empirical  study.  We  will  accordingly  refer  to  the  companion  report  many  times  throughout  this 
paper.  As  outlined  in  that  report,  our  primary  objectives  are  to:  (a)  identify  CRM-based  processes 
for  which  CMT  technologies  are,  or  can  be,  utilized  by  aircrews  and  other  mission  participants; 
(b)  document  evidence  of  training  effectiveness  using  these  processes  and  technologies;  (c) 
identify  the  characteristics  or  key  behaviors  emitted  by  the  most  effective  aircrews;  and 
(d)  recommend  opportunities  for  improving  CMT  through  more  effective  reinforcement  of  the 
key  behaviors.  The  concepts  discussed  in  that  framework  and  examined  in  this  report  concern  the 
performance  of  SOF  MC-130P  Combat  Shadow  aircrews  during  a  simulated  CMT  session.  How¬ 
ever,  we  believe  that  the  concepts,  methodologies,  and  results  of  the  present  effort  will  be 
applicable  to  other  weapon  systems  within  the  SOF  arsenal  as  well  as  other  USAF  assets. 

The  report  is  presented  in  five  major  sections.  The  remainder  of  this  first  section  describes 
how  CRM  training  and  research  has  evolved  during  the  past  few  years,  discusses  the  conceptual 
underpinmngs  of  crew  coordination  and  its  role  in  CMT,  defines  our  conception  of  tactical  team 
resource  management  (T^RM)  and  its  relationship  to  CRM,  presents  a  measurement  model  of 
T^RM,  and  re\  iew  s  some  of  the  tri-service  research  relevant  to  this  framework. 

In  the  second  section,  we  introduce  the  present  research  effort  by  discussing  our  research 
objectives,  hypotheses.  MC-130P  tactical  operations,  and  the  role  of  CMT  during  Annual 
Refresher  Training  (ART)  at  the  58  TRSS.  The  third  section  describes  the  research  methods, 
including  the  participants,  design,  data  collection  instruments,  assessment  procedures,  observa¬ 
tional  techniques,  and  the  resulting  data  structure. 

The  fourth  section  describes  the  empirical  results  from  the  present  effort,  focusing 
primarily  on  the  statistical  relationships  between  measures  of  crew  coordination  effectiveness  on 
the  one  hand  and  observed  mission  performance  on  the  other.  A  series  of  analyses  is  presented 
that  hierarchically  decomposes  the  overall  process-performance  relationship  into  its  constituent 
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elements,  in  which  subprocesses  and  mission  phases  are  considered.  Also  in  this  section,  we 
address  the  effect  of  mission  preparation  (MP)  on  mission  performance,  present  a  preliminary 
analysis  of  the  key  behaviors  associated  with  the  most  effective  aircrews,  and  assess  the 
influence  of  several  backgroimd  variables  (e.g.,  organic  crew  size)  on  mission  effectiveness.  The 
fifth  section  concludes  by  discussing  the  implications  of  the  present  results  for  conducting  further 
tactical  team  resource  management  (T^RM)  research,  applying  the  empirical  data  to  other  USAF 
research  and  development  (R&D)  areas  of  interest,  and  administering  CMT  within  the  58  TRSS. 

Evolution  of  CRM  Training  and  Research 

During  the  past  20  years,  CRM  has  become  a  widely  used  compionent  of  aircrew  training 
programs  for  both  the  civil  and  military  communities  (Gregorich  &  Wilhelm,  1993).  In  a 
landmark  study  of  the  effects  of  workload  on  aircrew  performance,  Ruffell  Smith  (1979)  reported 
that  the  behaviors  which  most  differentiate  effective  crews  from  weaker  ones  involve  leadership^ 
decision  making,  and  resource  management,  thereby  establishing  the  need  for  training  “softer” 
as  opposed  to  more  techmcally-oriented  skills.  Although  not  fully  accepted  by  crewmembers, 
researchers  believed  that  training  in  these  areas  would  yield  large  dividends  in  terms  of  increased 
flight  safety,  more  evenly  distributed  crew  workload,  and  more  efficient  communication. 

In  reviewing  the  evolution  of  CRM  training  and  evaluation,  three  trends  stand  out.  The  first 
pertains  to  operational  relevance.  Early  CRM  training  programs  failed  to  define  the  required 
aircrew  coordination  behaviors  in  operational  terms.  Rather,  CRM  was  viewed  as  a  loose  collec¬ 
tion  of  psychology,  leadership,  organization,  and  management  concepts  (A.  Diehl,  personal 
interviews,  November  1995).  The  behaviors  and  attitudes  that  were  trained  were  often  too 
touchy  feely  to  support  either  development  of  concrete  indices  or  to  be  accepted  by  aircrews 
as  necessary  for  flying  the  aircraft  (Helmreich,  1995).  Typical  CRM  topical  areas  included 
communication  processes,  team  building,  and  workload  management  (Gregorich  &  Wilhelm, 
1993),  where  it  is  not  immediately  clear  vsdiat  the  reinforceable  or  observable  behaviors 
associated  with  these  areas  would  be.  It  is  also  possible  that  such  training  might  not  be 
maximally  effective  in  a  more  tactically  oriented,  CMT  setting,  as  CRM’s  traditional  focus  on 
nontechnical  areas  would  be  overshadowed  by  tactical  and  combat  skill  requirements. 

In  response  to  these  criticisms,  both  the  airlines  and  military  attempted  to  add  more 
techmcal  meat  to  CRM  course  content.  Functional  areas  that  have  increasingly  found  their 
way  into  CRM  courses  cover  such  accident-related  topics  as  situation  awareness,  command 
authority,  resource  utilization,  and  operating  strategy.  Efforts  to  augment  the  technical  base  of 
CRM  course  offerings  are  still  underway,  with  more  work  clearly  needed  (Wilson,  1995). 

A  second  trend  concerns  the  specificity  of  CRM  training,  in  which  the  early  programs  were 
quite  generic,  with  all  airlines  and  airfiiame  types  receiving  similar  training.  Indeed,  a  “one  size 
fits  all”  strategy  characterized  the  initial  efforts  of  both  the  airlines  and  the  military.  While  the 
generic  nature  of  this  approach  proved  efficient  for  administering  large-scale  courses  and 
creating  widely  applicable  flightline  checklists,  its  effectiveness  has  not  withstood  empirical 
scrutiny  (Wilhelm,  1991). 
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Recently,  some  airlines  have  taken  a  more  problem-oriented  approach  in  which  a  given 
airline  explores  the  particular  CRM-related  problems  that  plague  its  operations  rather  than  taking 
a  global,  industry-wide  perspective.  Given  its  diverse  missions  and  multiple  major  commands, 
the  USAF  has  been  somewhat  slower  to  move  toward  weapon  system-specific  simulator  training 
and  the  corresponding  administration  and  assessment  of  CRM  course  materials  (D.  Wilson, 
personal  communication,  September  26, 1995). 

A  third  trend  is  evident  regarding  the  target  audience  for  CRM  training.  CRM  courses  in 
the  early  1980’s  attempted  to  improve  the  attitudes  of  aircraft  commanders  (ACs)  in  order  to 
promote  more  communication  and  information-sharing  in  the  cockpit.  CRM  training  in  the 
airlines  was  designed  to  impact  selected  captains  by  “fixing”  those  most  likely  to  resist 
information  from  copilots  (CPs)  or  other  crewmembers  in  time-critical,  high  workload  situations 
(Helmreich,  1995).  Similarly  within  the  USAF,  CRM  training  was  originally  focused  on 
individual  ACs.  Over  time,  coordination  concepts  have  been  expanded  to  include  other 
crewmembers,  where  it  may  ultimately  encompass  the  entire  mission  team,  including  intelligence 
(Intel),  tactics,  logistics,  weather,  airborne  command  and  control  (C&C),  air  traffic  control 
(ATC),  maintenance,  and  so  forth  (Andrews,  Bell,  &  Nullmeyer,  1995). 

Important  CRM  components  include  delineating:  CRM  principles  tied  to  operationally  rele¬ 
vant  behaviors,  the  appropriate  target  audience,  and  context-specific  effects.  Incorporating  these 
components  into  CRM  training  programs  tvill,  in  turn,  enable  researchers  and  training  specialists 
to  reinforce  specific  CRM  behaviors  for  particular  crewmembers,  increase  crewmember  and 
instructor  motivation  to  learn  and  apply  CRM  principles,  and  ultimately,  establish  an 
environment  for  determining  CRM  training  effectiveness. 

Despite  increasing  recognition  of  the  importance  of  CRM,  there  is  little  evidence  for  a 
direct  relationship  with  mission  performance.  This  is  particularly  true  in  the  context  of  CMT. 
While  there  is  a  vast  body  of  literature  concerning  aircrew  coordination,  only  a  small  subset  of 
that  has  addressed  CMT  (Silverman,  1994).  Moreover,  though  it  is  commonly  assumed  that 
effective  aircrew  coordination,  however  defined,  leads  to  improved  mission  performance,  very 
few  studies  have  demonstrated  an  empirical  link  using  tactically  realistic  training  scenarios.  One 
objective  of  the  present  effort  is  to  provide  such  evidence  while  avoiding  some  methodological 
pitfalls  that  have  plagued  previous  research  (Spiker,  Tourville,  Silverman,  &  Nullmeyer,  1996). 

Combat  Mission  Training  and  Tactical  Team  Resource  Management  (T^RM) 

CMT  for  the  MC-130P  is  an  integration  of  multiple  training  events  that  combine  to  transform 
an  aircrew  into  a  mission-ready  combat  team.  In  terms  of  ART,  CMT  is  a  combination  of  CRM 
academics  and  simulator  training,  technical  training  (e.g.,  emergency  procedures  (EPs)  and 
systems),  and  combat  tactics  (Wilson,  1995).  The  latter  covers  a  range  of  events,  such  as  threat 
recognition,  expendables  deployment,  mission  planning,  air  refueling  (AR),  low-level 
navigation,  and  covert  insertion  and  extraction.  It  is  this  added  emphasis  on  developing  tactical 
skills  that  makes  the  study  of  military  teams  so  different  from  its  commercial  counterpart, 
necessitating  the  development  of  customized  measurement  procedures  and  data  collection 
instruments. 
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The  traditional,  crew-level  conception  of  CRM  will  have  limited  applicability  to  the 
complex,  turbulent  CMT  environment.  This  view  was  substantiated  during  pilot  tests,  observa¬ 
tions,  and  interviews  conducted  with  SOF  subject-matter  experts  (SMEs)  in  the  early  stages  of 
this  project  (Spiker  et  al.,  1996).  On  the  basis  of  these  front-end  analyses,  we  have  adopted  a 
“tactical  team  resource  management”  or  T^RM,  model  as  a  logical  extension  of  the  CRM 
approach  discussed  above.  As  the  name  suggests,  T^RM  embraces  three  elements  that  distinguish 
it  from  CRM:  (a)  clear  delineation  of  a  combat  mission  team;  (b)  enhanced  focus  upon  tactical 
skills;  and  (c)  an  emphasis  on  managing  a  multitude  of  diverse  resotirces  within  a  dynamic 
environment.  Each  element  is  briefly  discussed  below. 

Tactical  Skills 

From  previous  studies  of  SOF  missions  (Spiker  &  Campbell,  1993;  1994;  Spiker,  1995), 
we  know  the  content  of  T^RM  must  contain  a  heavy  tactical  component  in  which  combat  skills, 
rather  than  communication,  attitudes,  and  interpersonal  relationships,  are  emphasized.  Successful 
conduct  of  a  SOF  operation  places  heavy  demands  on  the  tactical  skills  of  all  crewmembers, 
where  information  is  uncertain,  decisions  are  split-second,  and  timing  is  all-important.  Aircraft 
must  be  operated  at  low  altitude,  under  adverse  weather,  in  the  midst  of  relocatable  threats,  and 
with  no  margin  for  error  (Spiker  &  Campbell,  1993).  A  comprehensive  T^RM  approach  must 
ensure  that  reliable  measures  are  obtained  in  such  tactical  areas  as  navigation  accuracy,  threat 
avoidance,  meeting  control  times,  and  accomplishing  critical  mission  events.  The  specification  of 
the  requisite  tactical  skills  will,  accordingly,  be  highly  mission-  and  weapon  system-specific. 

Combat  Mission  Team 

A  primary  focus  of  the  T  RM  approach  is  its  emphasis  on  the  behavioral  processes  of  the 
entire  combat  team."  and  as  such,  is  intended  to  investigate  those  variables  that  both  measure 
and  predict  team  performance.  As  used  here,  a  team  is  composed  of  those  players  whose 
information,  actions,  and  decisions  impact  the  aircrew’s  mission  in  some  way  (Andrews  et  al., 
1995).  Besides  the  aircrew  itself,  this  includes  such  mission  participants  as  Intelligence  (Intel), 
planners,  weapons  and  tactics,  logistics,  weather,  airborne  C&C,  ATC,  maintenance,  as  well  as 
the  ground  "customers"  (Rangers,  SEALs,  etc.)  being  supported  by  the  Air  Force  Special 
Operations  Command  (AFSOC)  aircrew.  During  CMT,  actions  of  this  extended  tPiann  may  be 
role-played  b\  the  instructors  whose  inputs  are  scripted  to  promote  standardization  of  training. 
Such  scripting  is  ideal  from  a  research  standpoint  as  it  reduces  unwanted  variation  across  subject- 
crews.  A  comprehensive  T^RM  model  must  ensure  that  the  “data  hooks”  are  in  place  so  that  key 
interactions  between  players  within  this  extended  “team”  format  are  monitored  and  assessed. 

Resource  Management 


Within  traditional  CRM,  the  “resources”  that  are  to  be  managed  primarily  involve  informa¬ 
tion,  such  that  information-sharing  (e.g.,  between  AC  and  CP  or  pilots  and  other  flight  deck 
personnel)  is  encouraged.  Indeed,  early  CRM  work  focused  on  creating  a  cockpit  “climate”  in 
which  the  pilot  had  access  to  as  much  information  as  possible  concerning  the  safety,  status, 
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performance,  and  positioning  of  the  aircraft  within  the  larger  flight  environment  (Diehl,  1995). 
As  applied  to  SOF  operations,  the  T^RM  approach  takes  a  somewhat  broader  view  of  the 
resources  the  tactical  team  must  skillfully  manage.  In  addition  to  information,  the  SOF  team  is 
required  to  manage  its  most  precious  resource-time-to  ensure  that  all  required  mission  events 
are  performed  within  the  demanding  temporal  constraints  that  are  the  hallmark  of  SOF  missions. 
Another  scarce  resource  that  is  managed  within  our  T^RM  model  involves  the  allocation  of 
duties  among  a  limited  cadre  of  people.  Because  SOF  teams  must  leave  a  minimal  “footprint” 
when  performing  mission  events,  people  and  aircraft  are  kept  to  a  minimum  Thus,  effective 
T^RM  entails  utilizing  team  members  to  their  fullest.  This  allocation  is  not  only  important  during 
the  mission  itself,  it  is  also  imperative  during  mission  planning  (Spiker,  1995).  Consequently,  an 
effective  T^RM  model  must  accommodate  requirements  for  collecting  measurable  data  during 
the  all-critical  MP  stage. 


T^RM  Measurement  Model 

Conduct  of  T  RM  research  within  a  CMT  environment  should  be  guided  by  a  systematic, 
comprehensive  measurement  model.  Such  a  model  provides  a  common  language  to  define  the 
content  of  T^RM  functions,  establish  valid  indices  to  measure  T^RM  processes,  and  specify 
appropriate  procedures  for  collecting  T^RM  data.  Figure  1  depicts  the  T^RM  measurement  model 
(Spiker  et  al.,  1996)  used  to  guide  the  present  research. 


Figure  1 .  T^RM  Measurement  Model  (Spiker  et  al,  1996). 
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The  concepts  and  linking  arrows  in  the  figure  flow  fi-om  left  to  right,  reflecting  an  implicit 
timeline  (arrival  through  outbrief)  of  ART  activities.  Three  modules  feed  into  T^RM  Processes. 
The  first  two,  Crew  Background  and  Baseline  Attitudes,  reflect  the  fact  that  aircrews  vary  in  their 
background  experience  (e.g.,  squadron  affiliation,  hours  flown  as  a  crew)  and  attitudes  toward 
CRM.  To  capture  these  factors,  a  Crewmember  Background  Survey  (CBS  oval)  and  a  pre-Team 
Mission  Attitudes  Questionnaire  (TMAQl  oval)  were  administered.  However,  we  have  shaded 
the  TMAQl  (as  well  as  the  post-Team  Mission  Attitudes  Questionnaire  (or  TMAQ2)  oval)  to 
indicate  that  we  did  not  examine  these  factors  within  the  present  study. 

While  Training  Events  are  represented  as  a  single,  undifferentiated  component  in  the 
model,  we  recognize  that  the  crew  receives  many  salient  training  events  during  ART.  These 
include  CRM  academics,  relevant  technical  and  combat  tactics  training,  and  the  CMT  mission 
scenario  and  scripted  events  that  are  presented  before  the  crew  flies  in  the  simulator. 

The  five  subprocesses  depicted  in  the  T^RM  Process  module  represents  the  content  of  our 
approach.  They  were  selected  for  our  research  based  on  their  relevance  to  the  AFSOC  mission 
environment,  appropriateness  to  the  high  levels  of  experience  and  motivation  of  many  SOF  MC- 
130P  aircrews,  applicability  to  CMT,  and  amenability  to  measurement  by  outside  observers 
(Spiker  et  al.,  1996).  Where  possible,  we  attempted  to  identify  subprocesses  that  make  contact 
with  the  CRM  dimensions  identified  by  other  researchers.  The  five  T^RM  subprocesses  and  their 
formal  definitions  are  shown  in  Table  1. 

Table  1.  Definition  of  Five  T^RM  Subprocesses. 


Subprocess 

Definition 

Function  Allocation  (FA) 

• 

The  division  of  crew  responsibilities  so  that  workload  is  distributed  among 
the  crew,  avoiding  redundant  tasking,  task  overload,  and  crewmember 
disinterest  or  noninvolvement,  and  where  tasks  are  allocated  in  such  a 
manner  that  crewmembers  are  able  to  share  information  and  coordinate 
responsibilities. 

Tactics  Employment  (TE) 

All  the  analytic  activities  necessary  to  avoid  or  minimize  threat  detection  or 
exposure,  and  to  successfully  coordinate  complex  mission  events  and 
multiple  mission  objectives. 

Situation  Awareness  (SA) 

Maintenance  of  an  accurate  mental  picture  of  mission  events  and  objectives 
as  they  unfold  over  time  and  space. 

Command-Control-Communications 

(C3) 

Those  activities  required  to  involve  external  parties  in  the  mission  and  to 
maintain  communications  with  these  external  team  members;  communica¬ 
tion  within  the  crew;  and  controlling  the  sequence  of  mission  events 
according  to  the  mission  execution  plan. 

Time  Management  (TM) 

Ihe  ability  of  the  combat  mission  team  to  employ  and  manage  limited  time 
resources  so  that  all  tasks  receive  sufficient  time  to  be  performed  correctly 
and  critical  tasks  are  not  omitted. 

These  areas  are  depicted  in  the  white  T^RM  box  in  Figure  1  and  were  measured  using  our  Team- 
Mission  Observation  Tool  or  T-MOT  as  indicated  by  the  oval  above.  Since  we  know  that  these 
five  areas  do  not  encompass  the  entire  domain  of  what  would  properly  be  considered  T^RM  (or 
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even  CRM),  we  represent  Other  CRM  Processes  in  the  gray-shaded  box,  feeding  into  the  Team 
Mission  Performance  module.  These  processes  may  influence  team  mission  performance,  but 
were  not  measured  in  our  research. 

The  output  of  the  T^RM  Processes  module  feeds  directly  into  the  Team  Mission 
Performance  module.  Team  mission  performance  consists  of  those  indices  that  directly  result 
from  the  successful  (or  failed)  execution  of  important  T^RM  subprocesses  (e.g.,  TE,  FA).  In  our 
research,  team  mission  performance  is  reflected  in  such  indices  as  the  quality  of  the  pre-mission 
briefings,  completeness  of  the  navigation  chart(s),  and  instructor-supplied  ratings  of  how  well  the 
team  as  a  whole  and  individual  team  members  executed  each  phase  of  the  mission.  The  ovals 
above  the  Team  Mission  Performance  module  refer  to  two  tools  (Instructor  Rating  Instrument  or 
IRI  and  the  Team-Mission  Performance  Tool  or  T-MPT)  that  we  used  to  collect  this  information. 

Mission  outcome  consists  of  indices  that  reflect  whether  the  team  accomplished  its  stated 
mission  objectives.  When  performed  in  a  weapon  systems  trainer  (WST),  these  can  often  be 
recorded  by  computer.  Outcomes  include  airdrop  accuracy,  performing  airlands  within 
prescribed  time  windows,  on-ground  time  prior  to  airland,  and  minimi7.ing  (or  avoiding) 
exposure  to  threats. 

Mission  outcome  encompasses  the  criterion  environment  and  is  the  ultimate  yardstick  of 
crew  success  or  failure  used  by  the  operations  and  training  communities.  But  from  a  team 
research  standpoint,  solely  relying  on  outcome  is  risky  as  there  are  many  external  factors  which 
may  degrade  outcome,  yet  have  little  bearing  on  combat  team  effectiveness.  External  Factors 
(e.g.,  luck)  are  illustrated  in  the  gray  box  underneath  the  Mission  Outcome  module. 
Unfortunately,  when  assessing  team  coordination  imder  operational  conditions,  researchers  have 
little  control  over  these  external  factors.  As  such,  researchers  inherit  a  great  deal  of  noise  and 
uncontrolled  variability  in  their  outcome  measures,  making  it  difficult  to  infer  whether  good  or 
poor  team  coordination  has  occurred. 

The  Team  Mission  Performance  and  Mission  Outcome  modules  are  connected  with  a  bi¬ 
directional  arrow  to  indicate  that  some  indices  of  team  performance  are  based  on  data  that  would 
normally  be  used  to  measure  mission  outcome.  Within  our  model,  performance  leads  to  interim 
outcomes  that  in  turn  affect  subsequent  team  mission  performance.  For  example,  team  coordina¬ 
tion  affects  the  quality  of  the  mission  execution  plan,  which  in  turn  affects  subsequent  mission 
performance. 

The  Team  Mission  Readiness  module  reflects  the  end  state  of  the  crew  at  the  conclusion  of 
CMT.  Following  execution  of  the  mission  scenario  in  the  WST  and  a  formal  debriefing  by  the 
instructors,  the  aircrew  should  be  "transformed"  into  a  team  that  is  ready  to  perform  similar 
operational  missions.  While  such  transformations  are  not  observed  directly,  they  can  be  inferred 
from  behavior  changes  noted  by  trained  instructors  and  by  tracking  how  well  the  crew  performs 
upon  returning  to  its  operational  umt.  We  have  placed  an  unspecified  (shaded)  Measurement 
Instrument  oval  above  this  module  to  reflect  our  belief  that  one  can  measure  an  aircrew's  mission 
readiness,  though  such  measurement  was  not  part  of  our  current  research. 
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The  last  two  modules  are  Post-Mission  Attitudes  and  Crewmember  Self-Assessments.  They 
are  represented  in  parallel  to  reflect  the  fact  that  this  information  was  obtained  from  each  partici¬ 
pating  crewmember  using  the  TMAQ2.  By  comparing  crewmembers'  attitudes  toward  CRM  after 
academic  and  simulator  training  with  those  obtained  during  the  pre-ART  baseline,  researchers 
can  determine  the  degree  to  which  crewmember  attitudes  changed  over  the  course  of  CMT.  Yet  it 
is  important  that  one  not  mistake  a  relationship  between  training  events  and  CRM  attitude 
change  for  the  more  fundamental  relationship  between  team  coordination  processes  and  team 
mission  performance.  It  is  this  latter  relationship  that  has  the  greatest  implications  for  CMT  and 
is  the  focus  of  our  current  research. 

In  sum,  our  T^RM  measurement  model  embodies  a  number  of  features  that  are  critical  for 
establishing  direct  links  between  team  coordination  processes  and  mission  performance.  These 
include:  (a)  providing  multiple  measures  of  team  performance  in  addition  to  mission  outcome; 
(b)  assessing  a  series  of  tactically  relevant  coordination  processes;  (c)  measuring  team 
coordination  processes  and  performance  throughout  the  timeline  of  CMT  events;  and 
importantly,  (d)  emphasizing  independent  assessments  of  team  coordination  process  and  mission 
performance. 


Relevant  Tri-Service  Research 

A  particularly  useful  application  of  the  T^RM  model  is  to  organize  and  interpret  previous 
military  CRM  research.  As  is  evident  below,  the  knowledge  base  relevant  to  a  T^RM  approach  is 
limited.  The  purpose  of  this  review  is  to  give  the  reader  an  appreciation  for  the  types  of  issues 
that  have  been  studied  and  to  convey  the  urgent  need  for  empirical  data  that  establishes  the 
relationship  between  effective  team  coordination  and  mission  performance. 

Army 

Since  1990.  the  Army  has  been  conducting  research  to  assess  the  effectiveness  of  its  new 
Aircrew  Coordination  Training  (ACT)  program.  ACT  was  developed  in  response  to  aviator 
critiques  that  "existing  training  packages  for  crew  coordination  lacked  objective  standards  for 
measuring  crew  performance"  (Leedom,  1990,  p.  10).  Defining  crew  coordination  as  "a  set  of 
principles,  attitudes,  procedures,  and  techniques  which  transforms  individuals  into  an  effective 
crew"  (p.  10),  the  .^rmy  adopted  a  view  of  team  transformation  similar  to  that  in  our  T^RM 
model.  ACT  content  was  categorized  into  13  subprocesses  or  Basic  Qualities  (BQs),  such  as 
decision  techniques,  communication  acknowledgment,  and  information  transfer.  Each  BQ  is 
defined  by  a  set  of  performance  standards  and  evaluation  dimensions.  Instructor  pilots  (IPs)  use 
the  BQs  to  rate  aircreu  effectiveness  during  check  rides  much  as  they  assess  other  technical  tasks 
(Leedom,  1994). 

An  initial  research  study  addressed  the  issue  of  attitude  change — ^would  aviators  accept  the 
new  program?  Surveys  conducted  at  Fort  Campbell  showed  that  aviators  and  IPs  did  indeed 
exhibit  positive  changes  in  attitudes  toward  ACT  over  the  course  of  training  (Zeller,  1992). 
While  encouraging,  it  was  acknowledged  future  studies  are  required  to  substantiate  whether  ACT 
has  a  positive  impact  on  either  crew  mission  performance  or  mission  outcome  (Leedom,  1994). 
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One  such  study  looked  at  performance  of  UH-60  Black  Hawk  aviators  in  a  CMT  simulated 
mission  (Thornton,  Kaempf,  Zeller,  &  McAnulty,  1992).  Team  coordination  was  equated  with 
communication,  where  each  crew  was  assessed  in  terms  of  rate,  pattern,  content,  and  quality  of 
interactions  along  the  13  BQs  noted  above  (inquiry,  command,  declarative,  etc.).  Team 
performance  (or  mission  effectiveness)  was  measured  on  three  skills — ^navigation  accuracy 
(ground  track  deviations),  threat  evasion  (time  exposed),  and  nonprecision  instrument  approaches 
(SME-rating). 

Overall,  there  was  evidence  that  communication  was  significantly  related  to  some  mission 
effectiveness  indices.  For  example,  crews  who  were  more  successful  in  evading  threats  had  a 
pilot-flying  (PF)  who  issued  more  acknowledgments  than  a  PF  counterpart  in  unsuccessful 
crews.  Although  rate  of  communication  did  not  differ  significantly  among  poorly  performing 
crews  (i.e.,  those  who  navigated  inaccurately,  were  exposed  to  threats,  and  exhibited  poor 
approach  proficiency),  there  were  trends  suggesting  that  certain  types  of  communication  profiles 
were  consistently  related  to  positive  outcomes. 

The  investigators  concluded  there  is  some  evidence  for  a  direct  relationship  between 
aircrew  coordination  and  mission  performance.  However,  there  are  two  caveats  that  may  limit 
overall  applicability  of  their  results.  One,  they  observed  rather  low  levels  of  technical  proficiency 
on  basic  crew  tasks — map  interpretation,  terrain  identification,  radio  procedures— that  should 
have  been  mastered  prior  to  learning  CRM  skills.  Two,  their  definition  of  coordination  was,  from 
our  standpoint,  fairly  narrow  since  CRM  processes  other  than  communication  were  not  included. 

As  an  aside,  Thornton  et  al.  (1992)  provided  anecdotal  evidence  to  support  a  positive  rela¬ 
tionship  between  quality  of  mission  planning  and  mission  outcome.  Specifically,  they  noted  that 
crews  who  performed  better  instrument  approaches  had  spent  more  time  studying  the  approach 
plates  during  planning,  and  thus  needed  to  refer  to  them  less  often  during  the  high  workload 
landing  phase.  Such  evidence  encourages  the  CMT  timeline  depicted  in  Figure  1,  in  which 
measurements  of  team  coordination  encompass  the  entire  mission,  from  mission  planning 
through  execution  and  debrief 

Navy/Joint  Forces 

The  Navy  has  been  pursuing  an  aggressive  program  of  research  to  delineate  the  measurable 
aspects  of  team  process  and  performance.  Consistent  with  our  T^RM  model,  the  Navy  defines 
team  coordination  as  the  "essence  of  teamwork,  the  process,  the  moment-to-moment  behaviors, 
by  which  interdependent  team  members  achieve  important  goals"  (Brannick,  Prince,  Prince,  & 
Salas,  1995,  p.  641).  The  purpose  of  the  Brannick  et  al.  study  was  to  assess  the  validity  of  some  a 
priori  defined  dimensions  of  coordination  by  having  1 8  judges  independently  rate  a  large  number 
of  two-person  aircrews  fly  a  low-fidelity,  tabletop,  T-44  flight  trainer.  Judges  rated  crews  on  six 
process  dimensions— Assertiveness,  Decision  Making,  Adaptability,  SA,  Leadership,  and 
Communication — ^using  a  set  of  behaviorally  anchored,  5-point  rating  scales.  The  Navy  defined 
coordination  more  broadly  than  the  Army,  and  even  included  processes  (e.g.,  assertiveness, 
leadership)  excluded  from  our  model.  To  assess  performance,  an  IP  rated  each  crew  on  a  20-item 
checklist  (completing  flight  checklist,  making  required  altitude  corrections,  etc.).  Two 
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nontactical  mission  scenarios  (involving  point-to-point  flying)  were  flown  by  each  crew.  Half  of 
the  crews  consisted  of  current  Navy  T-44  pilots;  the  other  half  were  Navy  IPs. 

In  support  of  their  methodological  objectives,  Brannick  et  al.  demonstrated  consistency 
among  the  judges’  process  ratings  across  crews  and  scenarios.  They  also  found  a  fairly  strong 
relationship  between  the  various  team  coordination  process  ratings  and  overall  team 
performance,  with  correlations  ranging  from  r  =  .43  to  r  =  .69.  Comparison  of  the  process  ratings 
between  the  two  crew  experience  levels  revealed  a  surprising  lack  of  superiority  by  the  IPs,  with 
only  Assertiveness,  Decision  Making,  and  Commiuiication  approaching  signiflcance.  However, 
once  statistical  controls  are  imposed  to  correct  for  multiple  tests  of  signiflcance,  none  of  the 
mean  differences  proved  significant  (Spiker  et  al.,  1996).  The  lack  of  an  experience  effect  may 
have  been  because  the  non-tactical  mission  scenarios  were  not  sufficiently  challenging  to 
produce  substantial  performance  variation  across  crews.  Unfortunately,  comparisons  between  IPs 
and  trainees  on  the  expert-supplied  mission  performance  rating  were  not  reported. 

Recently,  the  Navy  has  been  expanding  their  team  process-performance  methodology  to 
distributed  interactive  simulation  (DIS)  environments.  To  that  end,  Dwyer,  Fowlkes,  Oser,  and 
Salas  (1996)  designed  a  set  of  instruments  to  separately  measure  critical  aspects  of  team  process 
and  team  performance.  In  one  test,  they  had  a  team  of  observer/controllers  (OCs)  observe  DIS 
training  for  five  days  of  Close  Air  Support  (CAS)  missions.  The  OCs  completed  the  process  and 
performance  instruments  in  addition  to  role-playing  the  higher  echelon  positions  needed  to  run 
the  scenario.  The  investigators  reported  that  their  instruments  were  reliable,  as  OCs  representing 
the  different  services  exhibited  similar  response  patterns.  They  also  found  that  teams  showed 
considerable  improvements  in  both  coordination  process  and  performance  from  Days  1  to  5, 
suggesting  a  pronoxmced  learning  effect. 

As  seen  through  our  measurement  model,  this  study  provides  a  rather  complete  picture  of 
team  performance  within  a  combat  mission  environment.  Indeed,  one  of  the  truly  admirable 
features  of  this  study  is  its  examination  of  team  coordination  in  a  much  larger  combat  team  envi- 
roiunent.  Other  praiseworthy  aspects  of  this  study  include:  on-line  data  collection  with  highly 
trained  observers,  collecting  team  process  and  performance  data  from  separate  sources,  collecting 
data  across  multiple  mission  phases  (planning,  contact  point,  attack),  and  using  a  realistic  tacticd 
scenario.  However,  the  authors  fail  to  report  any  performance  data  to  allow  exploration  of  the 
fimdamental  team  process-performance  relationship. 

In  a  follow-up  study,  Oser,  Dwyer,  and  Fowlkes  (1995)  observed  13  and  18  service  partici¬ 
pants,  respectively,  in  two  CAS-DIS  training  exercises.  As  before,  both  case  studies  found 
marked  improvement  in  team  process  and  team  performance  over  the  five  days  of  training 
Unfortunately,  process-performance  correlations  were  not  computed  which  would  have  shown 
which  aspects  of  team  coordination  would  impact  different  facets  of  mission  performance. 
Nevertheless,  these  studies  are  some  of  the  first  to  collect  systematic  team  performance  data  in  a 
DIS  training  environment,  and  should  therefore  help  further  the  knowledge  base  concerning  the 
training  strategies  that  can  be  employed  to  enhance  team  performance. 
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Air  Force 


In  an  early  study  of  crew  coordination,  Krumm  and  Farina  (1962)  investigated  the  impact 
of  integrated  simulator  training  on  B-52  mission  effectiveness.  The  investigators  collected 
process  data  on  the  pattern  and  rate  of  communication  between  crewmembers  during  selected 
segments  of  the  training  mission.  They  also  collected  objective  measures  of  navigation  and  bomb 
accuracy  performance.  They  found  that  the  method  of  training  had  a  positive  impact  on 
coordination,  as  crews  who  trained  together  had  better  communication  patterns  than  ones  who 
did  not.  Quality  of  communication  was  also  significantly  related  to  navigation  and  bombing 
accuracy.  For  example,  crews  who  navigated  more  accurately  also  volunteered  more  information. 

More  recently,  Povenmire,  Rockway,  Bunecke,  and  Patton's  (1989)  study  of  B-52  aircrew 
coordination  represents  one  of  the  strongest  attempts  yet  to  demonstrate  a  direct  relationship 
between  CRM  processes  and  mission  performance.  This  study  employed  a  number  of  the  meth¬ 
odological  features  we  have  discussed  in  our  T^RM  measurement  model,  ones  that  we  believe  are 
particularly  valuable  for  examining  process-performance  correlations  in  the  context  of  CMT. 

Povenmire  et  al.  observed  seven  intact  aircrews  fly  a  complex,  tactically  realistic  mission 
scenmo  in  a  high  fidelity  B-52  WST.  The  scenario  entailed  conducting  a  long-range  bombing 
mission  requiring  the  penetration  of  enemy  threats,  accurate  dropping  of  bombs,  and  intricate 
navigation  and  maneuvers.  SME  CRM  evaluators  assessed  aircrew  coordination  and  mission 
performance,  with  separate  raters  used  for  each.  Mission  performance  was  evaluated  on  three 
factors:  bombing  accuracy,  threat  avoidance,  and  technical  skill.  The  latter  consisted  of  a  number 
of  subfactors,  such  as  maintaining  appropriate  altitude,  navigating  accurately,  and  staying  within 
designated  control  times.  The  researchers  asked  the  evaluators  to  rank  order  the  crews  from  best 
to  worst,  based  on  their  subjective  impressions  of  the  three  mission  performance  factors. 

analysis  was  the  correlation  between  overall  aircrew  coordination  and  crew 
mission  performance  ranking,  where  a  strong  positive  relationship,  r  =  .84,  was  obtained. 
Povenmire  et  al.  then  compared  the  experts'  ratings  of  mission  performance  with  the  individual 
mission  outcome  factors.  Part-whole  correlations  showed  that  experts  primarily  used  bombing 
accuracy  to  judge  overall  mission  performance,  as  evidenced  by  a  correlation  of  r  =  .81.  The 
researchers  also  computed  a  series  of  part-whole  correlations  on  the  coordination  data  to 
determine  the  skill  dimensions  that  had  the  highest  loadings.  These  included  practicing  inquiry 
and  advocacy,  avoiding  distractions,  distributing  workload,  and  resolving  conflicts. 

Despite  the  simplicity  of  its  design  and  data  analysis  strategy,  the  Povemnire  et  al.  study 
stands  as  one  of  the  clearest  demonstrations  of  the  relationship  between  crew  coordination 
processes  and  mission  performance.  Indeed,  the  elegance  of  their  design  provides  unusually  clear 
insights  regarding  the  crew  coordination  processes  that  best  predict  mission  performance.  As 
discussed  in  the  next  section,  we  used  many  of  Povenmire  et  al.’s  methodological  features  in  our 
empirical  study  of  T^RM. 
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THE  PRESENT  RESEARCH  EFFORT 


This  section  provides  background  information  pertinent  to  the  present  research.  We  begin 
by  stating  our  research  objectives  and  the  hypotheses  the  study  was  designed  to  test.  Next,  we 
describe  MC-130P  operations  and  define  the  crewmembers  who  operate  the  aircraft.  We  close  by 
describing  how  ART  is  conducted  at  the  58  TRSS,  focusing  on  the  training  curriculum,  CMT 
scenario,  and  the  capabilities  of  the  MC-130P  WST. 

Study  Objectives 

The  objectives  of  the  present  research  were  established  in  accordance  with  three  levels: 
immediate,  near-term,  and  long-term.  The  immediate  objective  of  the  effort  was  to  demonstrate  a 
positive,  robust  relationship  between  team  coordination  and  mission  performance,  thereby 
replicating  the  basic  findings  of  Povenmire  et  al.  (1989).  With  that  relationship  established,  we 
then  wanted  to  identify  the  characteristics  (and  key  behaviors)  of  effective,  mission-ready 
aircrews  which  may  be  incorporated  into  training  program  procedures  and  technologies.  Another 
immediate  goal  was  the  development  and  validation  of  a  robust  set  of  data  collection  instruments 
that  can  be  applied,  with  modifications,  to  other  SOF  weapon  systems. 

In  the  near  term,  we  want  to  extend  the  present  research  and  conduct  a  more  controlled  test 
in  which  one  or  more  of  the  identified  interventions  are  packaged  as  a  set  of  training  program 
variables  whose  impact  will  be  assessed  using  a  traditional  control  versus  treatment  group 
experimental  design.  Candidate  training  package  enhancements  might  include: 

•  an  enhanced  CMT  simulator  mission  scenario, 

•  a  program  of  instructor  reinforcem?ent  of  key  CRM  behaviors, 

•  CRM  instruction  tailored  to  a  particular  weapon  system, 

•  selective  crewmember  cross-training  of  key  tasks  and  skills  (e.g.,  landmark  identification), 

•  customized  checklists  of  required  planning  activities, 

•  just-in-time  training  of  perishable  skills  (e.g.,  electronic  warfare  (EW)  tactics),  or 

•  deliberate  practice  of  key  mission  events  and  team  tasks. 

The  long-term  objective  of  the  research  program  entails  the  eventual  development  of  a  port¬ 
able,  research-ha.sed  instrument  that  identifies  the  team  elements  contributing  to  mission-ready 
aircrews.  Ideall\ .  such  an  instrument  would  give  squadron  commanders  the  capability  to  objec¬ 
tively  assess  the  readiness  of  assigned  crews  to  successfully  perform  a  given  mission.  Termed  the 
Team-Mission  Readiness  Assessment  Tool  (T-MRAT),  this  instrument  could  be  used  at  the 
squadron  level  b\  mission  commanders,  IPs,  and  Standardization/Evaluation  (Stan/Eval) 
personnel  to  assess  the  T^RM  standing  of  a  proposed  crew  complement.  The  T-MRAT  would 
quantify  a  crew’s  overall  likelihood  of  mission  success,  identify  weaknesses  in  a  crew’s 
composition,  specify  risks,  and  potentially  suggest  alternative  crew  configurations.  The 
assessment  tool  would  be  primarily  checklist  in  format,  could  be  administered  in  a  few  minutes 
under  austere  conditions,  and  would  be  scorable  on  the  spot.  The  role  that  the  T-MRAT  would 
play  within  our  T^RM  measurement  model  is  depicted  in  Figure  1  as  the  final  “Measurement 
Instrument”  oval. 
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Research  Hypotheses 

Five  major  hypotheses  were  derived  from  our  T^RM  measurement  model: 

1.  Team  coordination  processes  should  be  strongly  and  positively  related  to  mission 
performance.  Crews  who  exhibit  superior  coordination  behaviors  should  perform  better  during 
mission  execution  than  those  who  do  not.  If  found,  this  would  extend  the  Povenmire  et  al.  (1989) 
finding  with  B-52  Strategic  Air  Command  crews  to  another  weapon  system  and  USAF  major 
command.  As  described  in  Spiker  et  al.  (1996),  the  process-performance  relationship  is  the 
fundamental  tenet  of  the  proposed  research,  in  which  a  significant  finding  “gives  us  permission” 
to  probe  the  data  further,  to  test  the  other  hypotheses  stated  below. 

2.  The  various  T*RM  subprocesses  should  show  differential  relationships  to 
performance  across  mission  phases.  Although  we  expect  an  overall  process-performance 
relationship,  we  should  not  expect  every  T^RM  process  to  be  significantly  correlated  with  every 
phase  of  mission  performance.  We  do  not  believe  T^RM  to  be  a  single  entity.  Rather,  for 
performance  in  a  given  mission  phase,  only  one  or  possibly  two  T^RM  subprocesses  may  be 
statistically  significant  predictors.  For  example,  the  FA  subprocess  should  more  likely  have  a 
significant  impact  on  those  phases  of  performance  (e.g.,  low-level  navigation)  where  allocation 
of  crew  duties  can  substantially  reduce  overall  team  workload.  Such  selectivity  is  a  natural 
outcome  considering  that  different  phases  of  performance  will  require  different  types  of  crew 
involvement,  interaction,  and  decision-making.  While  most,  or  all,  the  process-performance 
correlations  across  phases  might  trend  positive,  only  1-2  subprocesses  should  reach  statistical 
significance. 

3.  The  quality  of  MP  should  be  strongly  and  positively  related  to  mission 
performance.  Simply  stated,  crews  who  do  a  more  thorough  job  of  MP  should  more  success¬ 
fully  execute  their  mission  than  those  who  do  not.  While  a  positive  relationship  between 
planning  and  performance  is  at  the  heart  of  all  military  MP,  there  is  surprisingly  little  empirical 
data  to  substantiate  this  all-important  assumption.  MP  and  its  effects  should  be  seen  both  as  a 
process  component— in  which  better  team  coordination  during  planning  should  lead  to  better 
mission  performance — and  as  a  performance  variable,  with  better  planning  products  (charts, 
briefings)  associated  with  superior  subsequent  mission  performance. 

4.  There  should  be  measurable  aspects  of  team  structure  that  are  statistically  related 
to  crew  effectiveness.  As  defined  by  an  amalgam  of  T^RM  coordination  and  mission 
performance  rankings,  the  more  “effective”  crews  should  have  some  structural  components  in 
common  that  are  not  present  in  the  less  effective  crews.  Predictive  structural  variables  might 
include  such  aspects  as  combined  crewmember  experience  in  the  area  of  operations  (AO),  hours 
flown  together  as  a  crew,  or  level  of  experience  for  key  crew  positions  (e.g.,  the  two  navigators). 
Alternatively,  salient  team  composition  variables  might  determine  how  crewmembers  are 
organized  (e.g.,  hierarchical  vs  informal)  or  how  information  is  shared  and  disseminated. 

5.  There  should  be  a  positive  relationship  between  the  crew’s  perceptions  of  T^RM 
effectiveness  and  mission  performance.  Crews  should  know  how  well  they  execute  the  various 
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T^RM  subprocesses  and  should  be  able  to  honestly  report  these  upon  completing  the  mission 
scenario.  In  general,  one  should  expect  that  crews  who  perceive  their  coordination  processes  to 
have  been  successful  will  also  report  having  performed  better  diuing  the  mission.  Although  such 
a  finding  lacks  the  independent  observation  of  process  and  performance  as  in  our  other 
hypotheses,  it  would  nonetheless  provide  supplemental  validation  of  the  relationship  using  an 
alternative  data  source. 

MC-130P  Tactical  Operations 

This  section  describes  MC-130P  Combat  Shadow  operations;  the  tactical  missions  executed 
and  the  crewmembers  who  fly  them.  Following  this,  a  description  of  ART  is  provided,  focusing 
on  the  present  training  curriculum,  the  CMT  simulator  scenario  used,  and  the  capabilities  of  the 
MC-130P  WST.  This  discussion  is  intended  to  give  a  general  overview  of  the  concepts  and 
terminology  used  in  SOF  operations. 

Missions 

In  the  broadest  terms,  the  primary  missions  of  the  MC-130P  Combat  Shadow  are:  (a)  AR 
support  of  SOF  MH-53J  Pave  Low  and  MH-60G  Pave  Hawk  helicopters,  (b)  aerial  delivery  of 
Special  Forces  (SF)  and/or  supplies,  and  (c)  airland  operations  to  austere  landing  zones.  Other 
SOF  missions  include  civil  affair  activities  (programs  to  control  or  influence  civilian  activity) 
and/or  psychological  operations.  The  environment  for  such  MC-130P  operational  missions  is 
typically  defined  as  a  moderate  threat  theater.  SOF  aircrews  typically  attempt  to  accomplish  their 
missions  using  night  vision  goggle  (NV G)  low-level  procedures  and  precise  timing  techniques  to 
meet  mission  objectives,  and  to  avoid  or  minimize  detection  by  hostile  forces.  Because  the 
taskings  of  the  MC-130P  are  so  varied,  it  is  difficult  to  construct  a  taxonomy  that  will  apply  in 
all  circumstances.  Thus,  the  abovementioned  mission  categories  serve  as  inexact  placeholders 
from  which  a  combat  mission  simulator  scenario  can  be  constructed. 

MC-130P  Combat  Shadow 


The  standard  MC-130P  aircrew  consists  of  seven  crewmembers.  Five  members  perform 
their  primary  flight  duties  in  the  cockpit:  the  AC,  CP,  Left  and  Right  Navigators  (LN  and  RN), 
and  FE.  The  Commumcation  Systems  Operator  (CSO)  and  Loadmaster  (LM)  crew  stations  are 
located  in  the  rear  cabin  compartment.  For  the  purposes  of  this  discussion,  the  LM  is  omitted. 
While  the  LM  receives  CRM  classroom  training  with  the  rest  of  the  crew  on  the  first  day  of 
ART,  he  is  trained  at  a  separate  site  for  the  rest  of  the  week  and  does  not  participate  in  the  CMT 
scenario.  This  is  primarily  due  to  the  lack  of  a  simulation  capability  at  the  58  TRSS  to  support 
the  training  of  LM’s  technical  and  CRM  skills.  A  general  description  of  the  other  crewmembers’ 
functions  follows: 

Aircraft  Commander  (AC).  The  AC  is  responsible  for  managing  the  overall  mission 
process  of  the  entire  aircrew.  Besides  flying  the  aircraft,  the  AC  monitors  the  progress  of  the 
crew  s  activities  as  they  work  to  accomplish  each  mission  objective.  The  actual  amount  of  work 
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performed  by  the  AC  will  vary  with  the  individual,  but  ideally,  the  AC  operates  as  the  filter  for 
all  communications,  and  leads  the  decisions  with  respect  to  the  mission’s  execution  details. 

Copilot  (CP).  The  CP  is  responsible  for  supervising  the  mission  process,  as  negotiated  and 
established  by  the  entire  aircrew.  The  CP  works  to  ensure  that  each  mission  detail  is  covered,  and 
alternative  courses  of  action  (COAs)  are  considered.  The  CP  participates  with  each  crewmember 
as  they  work  to  accomplish  each  mission  objective,  and  ideally,  integrates  those  individual 
activities  and  accomplishments  into  a  larger,  synergistic  mission  effect. 

Left  Navigator  and  Right  Navigator  (LN/RN).  The  LN  and  RN  are  responsible  for 
performing  the  many  calculations  required  to  support  aircraft  tactical  navigation  and  dynamic 
timing  and  control  activities  throughout  every  stage  of  the  mission.  These  duties  include,  for 
example,  the  development  of  tactical  low-altitude  flight  routing  to  reach  and  recover  from  the 
AO,  and  to  accomplish  all  major  mission  objectives  in  the  threat  environment. 

Flight  Engineer  (FE).  The  FE  is  responsible  for  determining  and  managing  the  aircraft’s 
performance  characteristics  (e.g.,  takeoff  and  landing  distance,  etc.).  This  information  typically 
includes  computing  the  aircraft  weight  and  balance,  field  density  and  pressure  altitude,  based  on 
environmental  conditions.  The  FE  is  also  responsible  for  management  of  the  aircraft’s  engines, 
fuel  systems,  and  electrical  power  systems,  so  that  other  crewmembers  may  concentrate  on 
developing  the  mission  scope  or  executing  the  mission  details.  In  addition,  the  FE  assists  in 
establishing  the  crew’s  Escape  and  Evasion  (E&E)  plan  in  the  event  the  aircraft  becomes 
disabled. 

Communication  Systems  Operator  (CSO).  The  CSO  is  primarily  responsible  for 
constructing  and  executing  the  communications  matrix  in  a  Communications  Execution 
Operating  Instruction  (CEOI)  format.  The  CSO  determines  the  radio  type  and  frequencies  to  be 
used  for  each  mission  phase  and/or  objective.  The  CSO  determines  the  C&C  net  to  be  employed 
with  the  other  mission  participants  and  ensures  that  the  required  coordination  issues  are 
established.  Additionally,  the  CSO  determines  the  extent  to  which  the  crew  can  conduct  then- 
tactics  plan  with  •  minimum  emissions,”  so  that  they  use  the  least  amount  of  en  route  radio 
communications  that  might  be  intercepted  by  hostile  ground  parties.  The  CSO  also  typically 
reviews  the  communications  capability  of  the  in-country  forces  to  determine  the  crew’s  Selected 
Area  For  Evasion  ( SAFE)  and  E&E  plans,  in  the  event  the  aircraft  becomes  disabled. 

MC-130P  Aircrew  Training  Devices 

The  primaiy  aircrew  training  device  used  for  MC-130P  ART  is  a  six-degree-of-ffeedom, 
high-fidelity  WST  based  on  the  SOF  Improvements  (SOFI)  version  of  the  MC-I30P  Combat 
Shadow  aircraft.  This  device  includes  a  CompuScene  V  image  generator  system,  a  fully 
correlated  Infrared  Detection  System  (IDS),  a  digital  radar  landmass  system,  navigation  systems, 
and  out-the-window  displays.  Navigation  systems  include  Doppler,  inertial  navigation  system 
(INS),  control  display  units,  and  a  central  computer  system  that  integrates  various  flight  systems. 
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The  MC-130P  Satellite  Navigation  System  (SNS)  is  a  part-task  trainer  used  to  train  MC- 
130P  navigators  and  CSOs.  This  device  may  operate  either  in  an  independent  or  integrated  mode. 
When  in  independent  mode,  the  device  operates  unconstrained  by  any  other  device  and  is  used  to 
train  either  the  navigators  or  the  CSO  in  crew  position-specific  tasks.  The  independent  SNS 
might  be  used  by  the  navigators,  for  example,  to  practice  high-altitude,  over, water,  celestial 
navigation  procedures,  or  by  the  CSO  to  practice  secure  communications  procedures. 

In  the  integrated  mode,  the  SNS  operates  in  conjunction  with  the  MC-130P  WST. 
Navigators  may  be  trained  in  either  the  SNS  or  WST  in  this  configuration,  while  the  CSO 
remains  in  the  SNS  for  orchestrated  training  with  the  students  in  the  WST.  This  is  the 
configuration  that  is  typically  used  \\^en  the  full  MC-130P  crew  is  tasked  to  conduct  mission 
operations  training.  The  pilots,  navigators,  and  FE  are  located  in  the  WST,  while  the  CSO  is 
located  in  the  SNS.  This  is  like  the  actual  SOFI  version  of  the  MC-130P  aircraft,  where  the  CSO 
station  is  geographically  separated  fi-om  the  flight  deck  in  the  rear  cabin. 

Annual  Refresher  Training  (ART) 

As  discussed  above,  ART  is  a  high-priority,  annual  training  requirement  for  SOF  MC-130P 
mission-qualified  crewmembers.  Content  of  this  training  is  specified  by  appropriate  regulations 
(i.e.,  AFSOCR  51-130:  Aircrew  Training),  and  the  training  matter  is  assumed  to  be  relatively 
constant  as  aircraft  systems  and  procedures  do  not  change  regularly.  The  ART  curriculum  at 
KAFB,  however,  has  undergone  major  shifts  with  regard  to  both  the  marmer  and  matter  of 
training  in  recent  years. 

Only  a  few  years  ago,  ART  consisted  mostly  of  academic  classroom  sessions  covering 
aircraft-specific  systems  and  EPs  reviews.  Following  each  half-day  classroom  session,  students 
were  then  required  to  apply  these  lesson  topics,  using  a  partial-task  training  approach,  in  the 
WST.  This  modular  training  method  worked  well  for  many  years,  and  those  USAF  units  using 
this  system  were,  for  the  most  part,  satisfied  with  the  traimng  provided.  There  was,  however,  a 
growing  consensus  in  the  relatively  small  SOF  community  that  the  expense  of  bringing  crews  to 
KAFB  each  year  for  a  repeat  of  the  same,  predictable  curriculum  was  becoming  prohibitive. 
Students  and  operational  squadrons  justifiably  recognized  the  capability  of  simulation  as  a 
training  medium  and  wanted  more  for  their  training  time. 

The  result  has  been  a  very  noticeable  shift  in  overall  training  philosophy  from  a  systems 
training  approach  for  individual  crewmembers  to  a  combat  mission  training  orientation  for 
full  mission  teams.  Systems  training  was  not  deleted  in  this  new  training  philosophy,  rather, 
systems  malfunctions  now  are  required  to  be  exercised  in  real  time  while  immersed  in  the 
complex  mission  environment.  Student  crews  performed  their  corrective  actions  to  instructor- 
induced  systems  malfunctions  and  EPs  while  other  live  stressors  also  inflicted  their  toll  on  the 
mission  team’s  actions. 

Coincident  with  the  shift  in  training  philosophy,  the  KAFB  aircrew  training  devices  and 
host  computers  underwent  significant  system  upgrades.  The  resultant  increases  in  simulation 
fidelity  afforded  new  and  unique  approaches  to  training  that  had  not  been  previously  possible. 
With  the  integration  of  EW  systems,  countermeasures,  improved  visual  systems  and  correlated 
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sensors,  among  others,  student  crews  could  now  perform  real-world,  operational  missions  in 
semi-immersion,  simulation  environments  with  a  significantly  increased  training  tempo. 

The  combination  of  improved  simulation  technologies,  a  strategy  for  implementing 
distance  learning  methods,  and  a  customer  thrust  for  a  mission-specific  and  an  operations 
training  curriculum,  naturally  led  to  implementation  of  a  CMT  refresher  philosophy.  The 
following  subsection  describes  the  general  sequence  of  curriculum  events  for  the  full  week  of 
ART. 


ART  Curriculum  and  Scheduling.  The  SOF  MC-130P  aircrew’s  ART  cumculum  is 
scheduled  for  five  full-day  sessions  of  training,  which  are  equally  divided  into  10  half-day 
training  periods  (see  Table  2).  The  first  half-day  of  training  provides  an  academic  review  of 
CRM  principles  and  anecdotal  discussions  of  recent,  CRM-related  experiences.  Following  this 
period,  the  student  crew  is  separated  into  functional  groups  (pilots  and  FE,  navigators,  and  CSO) 
to  join  their  respective  instructors  and  begin  their  week  of  aircraft-  and  mission-specific  training. 

Generally  speaking,  the  remaining  training  periods  are  grouped  into  academic  and 
simulator  training  sessions.  Each  day’s  instruction  begins  with  an  academic  period,  which 
focuses  on  a  review  of  aircraft  systems,  EPs,  and/or  mission  operations  training  using  a  variety  of 
multimedia  and  multidelivery  instructional  methods.  Following  each  academic  period,  students 
are  required  to  apply  the  preceding  academic  period’s  topical  matter  in  the  simulator  mission 
environment  (e.g.,  engine  fires  during  low-level  operations).  The  intent  is  to  provide  a  real-time 
stress  environment  for  these  crewmember  groups  so  they  may  rehearse  their  individual  and  crew 
responsibilities  and  responses  to  instructor-induced  malfunctions. 

Table  2.  Annual  Refresher  Training  Schedule 


1 

Academics:  CRM 

2 

Academics:  Systems  &  Procedures 

AC,  CP,  FE,  LN,  RN,  CSO,  LM 

AC,  CP,  FE  LN,  RN  CSO 

3 

Academics:  Systems  &  Procedures 

4 

Simulator:  Systems  &  Procedures 

AC,  CP,  FE 

LN,RN 

CSO 

WST;  AC,  CP,  FE  SNS;  LN.  CSO 

Academics;  S 

ystems  &  Procedures 

6 

simulator:  Systems  &  Procedures 

AC,  CP,  FE 

LN,RN 

CSO 

WST:  AC,  CP,  FE  SNS:  RN.  CSO 

■ 

Academics:  Systems  &  Procedures 

8 

Simulator:  Systems  &  Procedures 

AC,  CP,  FE 

LN,RN 

CSO 

MI:  AC,  CP,  LN,  RN,  FE  SNS:  CSO 

9 

Academics:  CMT  -  Preparation 

10 

l§imulator:  CMT  -  Execution 

AC,  CP,  FE,  LN,  RN,  CSO 

WST:  AC,  CP,  LN,  RN,  FE  SNS:  CSO 

CMT  Scenario.  As  illustrated  above,  several  daily  academic  review  and  simulator  practice 
and  reinforcement  sessions  are  provided  in  the  ART  curriculum.  The  last  training  day,  however, 
entails  a  capstone  project  that  is  devoted  to  full-crew,  mission-specific  combat  operations 
performance  in  a  mission-relevant  AO.  The  underlying  objective  of  this  real-time,  mission 
operations  training  is  to  enhance  overall  CRM  skill  and  performance.  However,  the  stated  intent 
is  also  to  let  the  crew  perform  at  a  significantly  higher  intensity  level  to  a  mission  tasking  that  is 
difficult,  complex,  and  relevant  to  their  real-world  mission  requirements. 
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This  simulator  mission  scenario  encompasses  a  complex  series  of  multiple  mission 
objectives  that  are  scripted  in  real  time  to  instructor-  or  crew-induced  problems.  Thus,  students 
are  tasked  in  this  training  to  accomplish  their  planning  and  execution  events  that,  in  the 
simulation  environment,  are  designed  to  push  them  to  near  overload  conditions.  The  subsections 
that  follow  describe,  in  linear  order,  the  unclassified  portions  of  the  CMT  scenario  that  were  used 
in  our  study,  along  with  the  corresponding  student  crewsi  activities  and  general  responsibilities  at 
each  step  in  the  training  day. 

Mission  Preparation  (MP).  The  mission  scenario  begins  with  an  extensive  MP  period. 
This  period  is  designed  to  enable  the  crew  to  formulate  a  mission  execution  plan  that  is 
compatible  with  their  capabilities  and  is  responsive  to  dynamic  mission  environmental 
conditions.  For  example,  a  typical  crew  might  use  this  period  to  prepare  several  mission  products 
in  response  to  the  scripted  mission  fragmentary  order  (FRAG).  These  products  might  include, 
among  others,  a  mission  flight  plan  that  contains  navigation  checkpoints  and  time  control 
markers,  an  AR  execution  plan,  and  a  Computed  Air  Release  Point  (CARP)  prediction 
worksheet. 

To  begin  the  period,  the  MC-130P  aircrew  arrives  at  the  mission  overview,  where  they  are 
provided  a  FRAG  brief  as  if  they  have  been  pulled  off  alert  status.  While  in  alert  crew  rest,  their 
scenario  has  been  evolving  and  their  particular  general  mission  plan  has  been  under  development 
by  a  dedicated  mission  planning  team.  The  simulated  time  is  approximately  four  hours  prior  to 
launch.  The  crew’s  tasking  is  to  “challenge”  these  planning  efforts  and  to  further  refine  the 
general  mission  plan  provided  to  them.  They  are  to  coordinate  all  requests  for  information  and 
relevant  logistical  details  into  an  integrated  mission  execution  plan  that  they  can  perform  as  the 
actual  mission  team. 

Instructors  are  tasked  to  role  play  C&C  elements  of  the  mission  tasking  team,  providing  the 
crew  with  a  general  siuiation  in-brief  and  periodic  mission  situation  reports  (SITREPs).  Specifi- 
cally,  the  initial  briefing  details  that  a  simulated  US  Army  platoon  has  reported  casualties  while 
conducting  mine-clearing  operations.  Five  additional  members  of  the  group  have  also  suffered 
disabling  injuries  and  require  immediate  medical  evacuation  (MEDEVAC)  to  a  forward- 
deployed  field  hospital.  Due  to  their  proximity  to  unfriendly  forces,  the  platoon’s  movement  is 
restricted,  and  they  are  requesting  immediate  MEDEVAC.  Two  MH-53J  Pave  Low  helicopters 
have  been  alerted  to  the  situation  and  are  tasked  to  transport  the  injured  to  the  field  hospital. 

The  MC- BOP’s  primary  mission  tasking  is  to  support  the  recovery  of  the  injured  personnel 
by  providing  AR  tanker  coverage  for  the  exfil  operation  and  MEDEVAC  flight.  To  support  this 
effort,  the  Shado  flight  is  also  tasked  to  insert,  via  a  personnel  CARP  airdrop,  several  pararescue 
jumpers  (PJs)  at  the  evacuation  Drop  Zone  (DZ).  The  intended  purpose  is  to  insert  an  SF  tpam 
who  will  prepare  evacuees  for  transport  and  prepare  the  landing  site  for  the  transload  operation. 

Secondarily,  the  MC-130P  crew  has  also  received  a  mission  tasking  to  transport  a  flag 
officer  and  staff  to  the  field  hospital.  This  hospital  is  located  on  an  airfield  that  is  presently  under 
joint  control  by  indigenous  army  and  rebel  factions.  The  scenario  calls  for  inserting  these 
personnel  under  the  semblance  of  a  diplomatic  negotiation  team  who  will  secure  airfield  landing 
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rights  before  the  inbound,  injured  personnel  arrive.  Additionally,  mobile  threat  systems  have 
been  recently  reported  in  the  surrounding  countryside,  and  these  should  be  considered  hostile. 
This  type  of  operation  dictates  the  requirement  to  perform  a  covert  approach  and  landing.  After 
inserting  the  team,  the  MC-130P  should  immediately  depart  the  airfield  and  continue  its  low- 
level  operations  and  tactics  to  recover  to  its  original  departure  location  airfield. 

The  students  are  handed  FRAG  worksheets  that  detail  these  mission  aspects,  and  provide 
certain  required  information  for  them  to  further  plan  their  mission  objectives.  Specifically,  they 
are  given  terminal,  route,  and  objective  area  weather  reports  and  detail  charts;  Operations  and 
Communications  Security  details;  Rules  of  Engagement;  and  Order  of  Battle  threat  details, 
among  others.  It  is  expected  that  the  students  will  integrate  these  materials  and  other  available 
information  sources  to  develop  an  executable  mission  plan. 

During  MP,  coordination  with  several  key  “players”  may  be  required  so  that  students  can 
gam  the  necessary  supporting  information  to  properly  prepare  the  logistics  portion  of  the  mission 
execution  plan.  To  facilitate  these  coordination  efforts,  instructors  role-play  the  functions  of 
Intel,  Tactics,  ATC,  maintenance,  and  weather.  Other  members  of  the  combat  mission  team  may 
exist  whose  tactical  actions  have  a  direct  effect  on  the  aircrew,  such  as  SF  on-board  “customers” 
and  other  AFSOC  aircraft  that  are  participating  in  this  mission  script  (e.g.,  the  MH-53J 
helicopters  that  are  to  be  refoeled  by  the  MC-130P).  Besides  the  supporting  and  tactical  players, 
those  individuals  and  functions  that  provide  a  C&C  role,  such  as  the  squadron/wing  command 
leadership  and  the  airborne  command,  control  and  communication  (ABCCC)  aircraft  may  be 
included  as  role  players. 

Table  3  presents  a  partial  listing  of  the  many  roles  and  responsibilities  of  each  crewmember 
during  MP.  The  duties  listed  are  not  absolute.  Depending  on  crewmember  experience,  expertise, 
or  interest,  these  duties  may  be  performed  by  other  crewmembers  or  perhaps  not  at  all.  The  table 
IS  mtended  to  illustrate  that  each  crewmember,  and  the  crew  as  a  whole,  is  heavily  invested  in 

activities  that  must  be  effectively  performed  and/or  coordinated  in  order  to  prepare  the  final 
comprehensive  mission  plan. 

The  remainder  of  the  MP  period  is  spent  conducting  several  pre-mission  briefings  that 
enable  the  crew  to  unite  as  a  cohesive  mission  team.  Once  these  briefings  have  been  completed  to 
the  crew  s  desired  lev  el  of  detail,  the  crew  should  be  focused  on  the  mission  objectives  and  the 
activities  by  which  these  objectives  will  be  accomplished. 

Mission  Execution.  Following  the  MP  period  (and  lunch),  the  crewmembers  enter,  as 
appropriate,  either  the  WST  or  SNS,  and  execute  their  mission  plan  with  little  or  no  direction  or 
assistance  provided  by  their  instructors.  The  instructional  premise  is  that  this  mission  is  to  be 
executed  tf  the  students  were  immersed  in  the  actual  aircraft  (e.g.,  every  checklist  will  be 
performed),  in  the  live  mission  environment  (e.g.,  all  threats  encountered  are  considered  as 
potentially  fatal  if  not  acknowledged  in  a  timely  and  appropriate  manner).  Responses  to  any  self- 
induced,  instructor-induced,  or  scripted  stimulus  condition  are  to  be  responded  to  in  real-time, 
and  if  that  response  results  in  an  undesirable  situation  or  condition,  the  crew  must  live  with  their 
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decisions.  Students  are  briefed  that  this  mission  is  to  be  given  the  highest  priority  of  execution, 
with  overall  mission  success  as  the  ultimate  goal. 

Table  3.  Crewmember  Roles  and  Responsibilities  During  Mission  Preparation 


Crewmember 

Mission  Preparation 

Aircraft  Commander 
(AC) 

Reviews  the  mission  tasking  to  provide  an  initial  risk  assessment. 

Accepts  the  crew's  mission  plan  and  prepares/coordinates  the  mission  execution  briefmg. 
Prepares/coordinates  requests  for  “outside”  agencies  to  provide  necessary  logistics 
support. 

Copilot  (CP) 

Reviews  mission  details  to  ensure  that  all  CO  As  are  considered  and  the  mission’s  risk 
assessment  is  correct. 

Woilcs  with  the  RN  to  validate  threat  capabilities  and  establish  a  tactics  execution  plan. 
Checks  the  FE’s  Takeoff  and  Landing  Data  (TOLD)  and  other  computations. 

Left  Navigator  (LN) 

Prepares  the  Flight  Chart  with  all  annotations  necessary  for  low-level,  NVG  flight 
operations  (e.g.,  route  of  flight,  waypoint  information,  terrain  obstructions,  etc.). 

Organizes  the  mission  data  to  facilitate  programming  of  the  aircraft’s  mission  computers. 

Right  Navigator  (RN) 

Performs  threat  capabilities  analyses  and  prepares  the  tactics  execution  plan. 

Computes  the  CARP  worksheet  and  prepares  the  airdrop  plan. 

Prepares  the  AR  rendezvous,  join-up,  and  contact  plans;  pro]X)ses  alternative  COAs  in  the 
event  the  AR  is  unsuccessful. 

Flight  Engineer  (FE) 

Computes  the  aircraft  fuel  management  plan. 

Computes  the  aircraft  weight  and  balance  data,  and  TOLD  cards. 

Assists  in  preparing  an  aircraft  destruction  plan  in  the  event  the  aircraft  is  disabled  in 
hostile  territory. 

Communication 

Systems  Operator 
(CSO) 

Prepares  the  Communications  Execution  Operating  Instruction  (CEOI)  in  response  to  the 
threat  environment. 

Resolves  C&C  communications  security  matrix  needed  to  coordinate  all  mission  activities. 
Determines  the  SAFE  and  plans  the  E&E  plan  in  the  event  the  aircraft  becomes  disabled  or 
is  downed  in  hostile  territory. 

Since  students  may  employ  different  strategies  in  planning  their  mission,  the  mission 
execution  period  may  be  envisioned  for  our  purposes  as  the  integration  of  smaller,  discrete 
mission  phases  composed  of  several  operational  objectives.  Within  each  of  these  phases,  each 
crewmember  is  required  to  perform  a  series  of  complex  tasks.  Moreover,  the  crew  must  perform 
a  series  of  difficult  duties  as  a  collective  unit  to  ensure  the  successful  accomplishment  of  the 
particular  mission  objectives.  Below,  we  describe  each  of  these  mission  phases,  illustrating  the 
means  by  which  each  crewmember,  and  the  collective  crew,  might  work  to  successfully 
accomplish  these  objectives.  Our  purpose  here  is  to  paint  a  general  picture  of  how  the  crew  might 
execute  the  simulated  CMT  scenario  they  have  prepared  for  during  the  previous  period. 

The  first  mission  execution  event  is  the  Low-Level  (LL)  tactical  operations  phase.  The 
mission  objective  is  to  conduct  NVG  low  altitude  flight  en  route  using  proper  tactical  mission 
management  procedures  (e.g.,  very  low  altitude  flight,  high-speed  maneuvering,  terrain  masking, 
optimal  tactical  routing)  for  the  threat  environment.  In  this  phase,  the  AC  physically  flies  the 
aircraft  using  NVG  low-level  altitudes  and  procedures  along  the  intended  route  of  flight,  under 
the  steering  guidance  provided  by  the  LN.  The  CP  assists  the  LN  with  route  navigation  by 
confirming  the  NVG  visual  reference  points  that  have  been  identified  from  the  mission  charts. 
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navigation  systems  information,  and  digital  scan  radar  and/or  IDS  presentations.  The  CP  also 
assists  the  AC  by  continually  cross-checking  the  relevant  flight  instruments  for  proper  altitude, 
airspeed,  and  attitude  indications,  and  advising  the  AC  of  unsafe  deviations  and  corrective  action 
where  necessary. 

In  this  mission  phase,  the  AC  and  CP  are  the  “eyes  outside”  to  the  visual  world  for  the  LN 
who  is  not  on  NVGs.  In  that  respect,  the  LN  maintains  a  continual  dialogue  with  the  AC  and  CP 
to  confirm  that  the  pilots  are  communicating  what  is  expected,  and  that  they  are  flying  to  the 
correct  visual  reference  point.  Additionally,  the  LN  works  to  overcome  a  major  known  NVG 
limitation,  distance  estimation  error,  by  constantly  “calibrating”  the  pilot’s  eyes  on  their 
distances  from  these  visual  reference  points. 

During  the  LL  phase,  the  RN  is  primarily  responsible  for  continual  assessment  of  the 
overall  threat  situation,  and  informing  the  crew  of  or  performing  those  actions  required  to  defeat 
a  threat  encoxmter.  To  that  purpose,  the  RN  coordinates  with  the  LN  for  alternative  steering 
commands  away  from  threat  locations,  and  with  other  crewmembers  on  the  performance  of  a 
threat  maneuver  or  a  weapons  flyout  status.  It  is  very  likely  that  the  crew  will  encounter  an 
unplanned  threat  during  this  mission  phase.  Even  if  the  mission  tactics  plan  was  prepared 
perfectly,  and  the  crew  was  able  to  avoid  or  successfully  mask  from  all  known  threats,  the 
instructors  will  often  program  unscripted  threats  into  the  flight  path  to  force  the  crew’s 
spontaneous  response  to  a  hostile  intent  or  weapons  flyout. 

It  is  also  possible  that  the  crew  may  incorrectly  apply  their  planned  tactics  in  this  situation, 
perhaps  suffering  aircraft  damage  as  a  result.  In  this  case,  the  FE  is  an  “eyes  inside” 
crewmember  who  keeps  the  aircraft  flying  from  an  engines  and  power  perspective,  and  advises 
the  AC  of  malfunctions,  battle  damage  and  effect,  and  any  corrective  actions  necessary.  The  AC, 
CP,  and  LN  continue  to  coordinate  those  activities  necessary  to  fly  the  aircraft  out  of  further 
harm,  and  yet  continue  toward  the  next  mission  object.  The  RN  works  to  coordinate  those  tactics 
necessary  to  beat  the  threat,  while  the  FE  keeps  the  aircraft  flying.  All  the  while,  the  CSO  is 

communicating  with  several  C&C  elements  to  keep  them  appraised  of  the  current  situation  and 
mission  progress. 


This  phase,  as  well  as  the  others,  also  includes  several  scripted  occurrences  of  irrelevant 
communications  and  unforeseen  events  that,  like  the  real  world,  occur  in  flight.  The  scenario’s 
intent  is  to  insert  a  number  of  distracting  events  throughout  the  mission  that  requires  the  CSO 
(and  others)  to  obtain  and  filter  relevant  information,  determine  the  mission  impact,  and  solicit 
alternative  COAs  in  response  to  these  changed  mission  conditions.  For  example,  one  instractor 
might  role-play  the  ABCCC  command  element,  and  report  that  one  of  the  helicopters  expected 
for  the  AR  has  ditched  while  enroute  to  the  objective  area.  This  would  require  the  crew  to 
determine  the  degree  to  which  this  change  impacts  their  primary  mission  activities,  or  whether 
they  are  required  for  further  on-scene  Search  and  Rescue  support.  Several  outcomes  are  possible, 
and  the  crew  must  work  the  issue  through  to  its  correct  conclusion.  If  an  incorrect  conclusion  is 
produced— for  example,  the  crew  decides  it  is  appropriate  to  divert  from  its  original  mission 
tasking— the  instructor  might  indirectly  intervene  to  steer  the  crew  back  to  the  original  mission 
objective  by  role-playing  ABCCC’s  refusal  to  grant  their  mission  divert  clearance. 
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The  next  mission  phase  is  Air  Refueling  (AR).  In  this  phase,  the  mission  objective  is  the 
successful  conduct  of  tactical  in-flight  AR  operations  for  multiple  MH-53  J  Pave  Low  helicopters 
within  prescribed  time,  course,  and  altitude  constraints  in  a  threat  environment.  Again,  the  AC  is 
physically  flying  the  aircraft  in  order  to  execute  the  AR  operation’s  rendezvous,  join-up,  and 
refueling  procedures.  The  AC  is  accepting  the  direct  guid^ce  and  control  of  the  LN  who  is 
providing  continuous  steering  commands  based  on  navigation  systems  information  and/or  radar 
presentations,  and  the  RN’s  regular  interpretation  of  IDS  images.  Meanwhile,  the  CP  continues 
to  provide  backup  support  for  these  activities,  the  FE  prepares  the  aircraft  for  the  fuel  transfer 
operation,  and  the  CSO  provides  constant  status  communications  with  the  C&C  elements  and/or 
the  helicopter  parties.  Upon  completion  of  the  AR  operation,  the  aircraft  is  reconfigiued  for  low- 
altitude  flight  operations  and  high-speed  maneuvering  requirements  to  meet  the  next  mission 
objective. 

The  next  mission  phase  is  the  Airdrop  (AD)  operation.  The  objective  of  this  mission  phase 
is  to  successfully  conduct  the  CARP  airland  of  SF  personnel  within  prescribed  time,  course,  and 
altitude  constraints  in  a  threat  environment.  In  this  phase,  the  AC  is  required  to  fly  the  aircraft 
with  exacting  altitude,  airspeed,  and  heading  parameters,  as  provided  by  the  LN,  in  order  to 
correctly  position  the  aircraft  for  the  AD  operation.  The  CP  assists  the  AC  in  this  phase  by 
continually  providing  cross-checks  of  the  aircraft  attitude,  and  advising  of  deviations  and 
corrective  action  where  necessary.  The  CP  also  monitors  outside  visual  references  to  help 
identify  the  DZ,  and  determine  the  CARP  offset  aim  points  for  run-in.  The  LN  and  RN  are  tuning 
and  interpreting  the  digital  scan  radar  presentation  and  IDS  object  sensitivity  in  order  to  identify 
the  DZ,  as  well  as  configuring  the  navigation  system’s  sensitive  (e.g.,  “hot  cursor”)  steering 
commands.  The  FE  configures  the  aircraft  for  the  AD  operation  (e.g.,  opens  appropriate  doors  so 
that  the  simulated  SF  personnel  may  leave  the  aircraft),  and  configures  certain  systems  to 
minimize  damage  if  they  fall  under  hostile  fire  while  “exposed”  in  the  AD  operations’  low 
altitude  and  slow  airspeed  requirements.  The  CSO  coordinates  for  the  final  drop  clearance  and 
relays  this  information  to  the  crew  for  the  final  drop  decision  which  is  made  by  the  LN  and  AC. 
Upon  completion  of  the  drop  operation,  the  aircraft  is  reconfigured  to  resume  the  high-speed, 
low-level  operation  previously  described  in  order  to  reach  the  next  mission  objective. 

The  crew  prepares  to  perform  the  final  Infil/Exfil  (I/E)  operation  required  to  transload  the 
flag  officer  and  the  diplomatic  negotiation  team  at  the  forward-deployed  field  hospital  landing 
zone  (LZ),  and  to  evacuate  their  aircraft  fi-om  the  unsecured  airfield.  The  objective  of  this 
mission  phase  is  the  successful  conduct  of  covert  insert  and/or  extraction  at  tactical  landing  sites 
for  transload  purposes  within  prescribed  time,  course,  and  altitude  constraints  in  a  threat 
environment. 

In  this  phase,  the  AC  is  responsible  for  correctly  positioning  the  aircraft  for  a  Self- 
Contained  Approach  (SCA),  where  the  on-board  navigation  systems  are  programmed  using 
unique  techniques  to  provide  semi-precise,  initial  approach  path,  final  approach,  and  missed 
approach  navigation  steering.  The  approach  steering  is  coordinated  with  the  LN,  who  interprets 
navigation  information  in  the  form  of  verbal  steering  commands  that  are  confirmed  on  the  flight 
instruments  by  the  CP.  The  RN  identifies  the  LZ  aim  point  using  the  IDS  presentation,  and 
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confirms  the  LN’s  interpretation  of  the  radar  picture.  The  FE  is  the  coordination  link  for  the 
timely  completion  of  all  required  checklist  items  and  responses  and  monitors  the  final  approach 
and  landing  speeds  while  confirming  the  aircraft’s  configuration  for  landing.  Finally,  the  CSO 
coordinates  the  required  final  approach  and  landing  clearances,  and  the  transload  procedures  with 
any  ground  parties.  Upon  completion  of  the  transload  operation,  the  aircraft  is  reconfigured  for 
immediate  takeoff,  and  resumes  the  low-level  operation  to  the  final  recovery  airfield. 

Table  4  provides  a  summary  of  these  general  mission  execution  roles  and  responsibilities. 
Again,  these  duties  are  not  absolute  and  may  vary  depending  upon  crewmember  experience, 
expertise,  or  interest.  As  with  Table  3,  Table  4  is  intended  to  show  that  each  crewmember  is 
heavily  involved  in  executing  certain  activities  that  must  be  collaboratively  and  effectively 
performed  in  order  to  accomplish  the  mission  tasking. 

Table  4.  Crewmember  Roles  and  Responsibilities  During  Mission  Execution 


Crewmember 

Mission  Execution 

Aircraft  Commander 
(AC) 

Conducts  each  mission  event  briefing  to  coordinate  or  update  objective  details. 

Coordinates  the  separate  activities  of  each  crewmember  into  a  integrated  unit  focused  on 
objective  accomplishment  in  an  environment  of  dynamic  changing  conditions. 

Provides  final  Go  or  No-Go  decisions  to  continue  operations. 

Copilot  (CP) 

Assists  in  interpreting  flight  chart  annotations  necessary  for  low-level  NVG  flight 
operations. 

Promotes  the  activities  of  each  crewmember  as  a  cohesive  imit  focused  on  successful 
objective  accomplishment  in  an  environment  of  dynamic  changing  conditions. 

Left  Navigator  (LN) 

Reads  and  interprets  the  Flight  Chart  annotations  necessary  for  low-level  NVG  flight 
operations  (e.g.,  correlating  map  features  with  radar  or  IDS  presentations). 

Commands  the  Infil/Exfil  SCA  operation. 

Enters  the  mission  data  to  program  aircraft  mission  computers. 

Right  Navigator  (RN) 

Performs  continuous  threat  capabilities  analyses  and  executes  the  tactics  plan. 

Re-computes  the  CARP  and  updates  the  airdrop  plan  for  the  LN. 

Coordinates  updates  to  the  AR  rendezvous  and  join-up  plan  with  the  LN  in  response  to 
dynamic  condition  updates. 

Flight  Engineer  ( FF. ) 

Executes  the  aircraft  fuel  management  plan. 

Re-computes  the  aircraft  weight  and  balance  data,  and  TOLD  cards  after  each  mission 
objective  is  completed. 

Coordinates  completion  of  checklist  items  and  ensures  responses  are  verified. 

Communication 

Systems 

Operator  (CSO) 

Executes  the  CEOI  in  response  to  the  threat  environment  and  mission  conditions. 
Continually  monitors  the  mission  progress  and  updates  the  E&E  plan  as  necessary. 

Filters  mission  information  between  the  flight  crew  and  C&C. 

METHOD 

This  section  describes  the  research  methods  and  procedures  that  we  used.  First  we  cover  the 
backgrounds  of  the  SOF  crews  who  served  as  research  participants.  Second,  we  present  the  logic 
underlying  the  quasi-experimental  design  that  was  used.  Next,  we  discuss  the  instruments  that  we 
used  to  collect  the  empirical  data.  We  then  describe  the  procedural  steps  that  were  followed  in 
collecting  data  firom  four  primary  sources:  student  questionnaires,  instructor  questionnaires, 
researcher  structured  observations,  and  instmctor/operator  station  (lOS)  pages. 
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Participants 


Thirteen  MC-130P  SOF  aircrews  (79  crewmembers  total)  were  observed  during  their  week- 
long  visit  to  KAFB  for  ART.  Observations  were  made  over  an  8-month  period  (March-October 
1996).  Two  of  these  crews  (Crews  #1  and  #8)  were  eliminated  from  subsequent  analyses  because 
they  were  unable  to  fly  the  simulated  mission  due  to  simulator  malfunctions.  Thus,  1 1  crews  (67 
crewmembers)  were  included  in  the  analysis. 

The  typical  make-up  of  an  MC-130P  crew  for  ART  consists  of  six  members:  two  pilots, 
two  navigators,  one  FE,  and  one  CSO.  In  this  study,  crew  size  varied  from  5-7  members  because 
two  crews  trained  with  an  extra  CSO  for  the  observed  mission  while  one  crew  did  not  have  a 
CSO  at  all.  However,  the  modal  crew  size  was  six.  The  distribution  of  crew  size  across  the  1 1 
participating  crews  is  displayed  in  Table  5. 


Table  5.  Crew  Size  of  Each  Participating  MC-130P  Crew 


Crew  No. 

Crew  Size 

2 

6 

3 

6 

4 

7 

5 

7 

6 

6 

7 

6 

9 

6 

10 

6 

11 

5 

12 

6 

13 

6 

As  shown  in  Table  6,  the  participants  were,  on  average,  very  experienced  SOF  personnel. 
Nevertheless,  four  crewmembers  had  only  100-200  hours  prior  flying  experience  in  the  MC- 
130P,  with  one  crewmember  reporting  zero  hours.  (This  individual  was  a  highly  experienced 
pilot  vrith  1,550  hours  in  MC-130H  models  who  was  just  “filling  in”  for  the  week  with  the  MC- 
130P  crew.)  However,  none  of  these  inexperienced  crewmembers  served  in  the  same  crew 
during  the  study. 


Table  6.  Crewmember  Flying  Experience  Summary 


Frequency 

Mean 

Minimum 

Maximum 

Standard 

Deviation 

Age 

67 

34 

25 

48 

5 

MC-130P  Flying  Hours 

67 

1286 

0 

7550 

1231 

Total  Flying  Hours 

67 

3056 

500 

7550 

1231 

CRM  Training  Hours 

67 

37 

0 

200 

37 
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In  addition,  the  crewmembers  within  the  various  crews  hailed  from  different  SOF 
squadrons  as  indicated  in  Table  7,  with  both  enlisted  and  officer  ranks  well  represented  (see 
Table  8).  Appendix  A  presents  a  detailed  demographic  summary  for  each  crew.  These  factors 
included:  age,  MC-130P  flying  hours,  total  flying  hours,  CRM  training  hours,  hours  of 
experience  in  the  mission  AO,  distribution  of  ranks,  and  squadron  affiliation. 

Table  7.  Squadron  Affiliations  of  the  Participating  MC-130P  Crewmembers 


Squadron 

Frequency 

Odier 

6 

5th  SOS 

10 

9th  SOS 

16 

67th  SOS 

12 

17th  SOS 
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Table  8.  Distribution  of  Participating  MC-130P  Crewmembers  by  Rank 


Rank 

Frequency 

Ltc 

3 

Maj 

6 

Capt 

35 

Msgt 

5 

Tsgt 

7 

Ssgt 

10 

Amn 

1 

Highly  experienced  Mission  Training  Support  System  (MTSS)  MC-130P  instructors  also 
participated  in  this  effort  as  supplemental  performance  evaluators.  Bach  crew  position  was  paired 
with  an  instructor  during  the  entire  week  of  ART,  and  this  instructor  was  present  during  Day  5 
training.  Following  observation  of  the  Day  5  mission,  every  instructor  completed  an  IRI  whose 
specifics  are  described  later. 


Design 

Five  features  were  considered  central  to  the  design  of  the  present  study: 

(1)  quasi-experimental  observation  method; 

(2)  independent  assessments  of  team  coordination  and  mission  performance; 

(3)  robust  team  assessment  measures; 

(4)  multimeasure,  multimethod  (MM-MM)  mix  of  variables;  and 

(5)  behaviorally  anchored  ratings  scales  (BARS). 
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Quasi-Experimental  Observational  Approach 

The  present  study  collected  data  in  a  naturalistic  observational  setting.  There  were  several 
reasons  for  this  choice.  First,  we  were  able  to  capitalize  on  ongoing  MC-130P  ART  using  a 
combat  mission  scenario  that  was  already  in  place.  We  worked  with  the  AFSOC  training 
community  on  a  not-to-interfere  basis  and,  as  a  result,  had  access  to  a  highly  experienced, 
volunteer  subject  pool.  Second,  use  of  a  naturalistic  observation  paradigm  offered  the  advantages 
of  operational  relevance  (external  validity)  and  a  clear-cut  application  of  team  mission 
performance  and  coordination  principles.  Third,  this  approach  provided  us  with  the  ability  to 
immediately  fold  back  lessons  learned  into  the  58  TRSS  ART  program,  without  the  lag  time  so 
often  associated  with  laboratory  research  efforts. 

Although  we  did  not  explicitly  manipulate  any  experimental  variables,  we  were 
nevertheless  able  to  examine  key  relationships  by  using  post  hoc  groupings  of  the  data  we 
collected.  For  example,  we  examined  the  effect  of  a  crew’s  predominant  squadron  affiliation  on 
team  mission  effectiveness.  Consequently,  some  of  analyses  reported  herein  are  based  on  a  quasi- 
experimental,  as  opposed  to  a  strictly  correlational,  approach. 

Independent  Assessments  of  Team  Coordination  and  Mission  Performance 

Independent  assessments  of  team  coordination  processes  and  team  mission  performance 
were  also  essential  to  the  present  approach.  Specifically,  independent  collection  of  coordination 
data  fi-om  one  researcher  and  mission  performance  data  from  a  second  researcher  helped  ensure 
the  validity  of  direct  comparisons  and  avoided  the  artificially  inflated  correlations  that  stem  from 
obtaining  both  sets  of  measures  from  the  same  rater.  Using  two  individuals  to  make  separate 
assessments — in  conjunction  vvdth  selected  computer  printouts — also  enabled  detailed  accounts 
of  both  coordination  process  and  performance  behaviors.  Each  rater  was  able  to  focus  all  of 
his/her  attention  on  their  assigned  dimensions. 

A  highly  experienced,  former  AFSOC  MC-130P  operator  was  employed  to  collect 
coordination  data;  a  second  researcher  (Ph.D.  psychologist)  and  four  ART  instructors  assessed 
mission  performance.  lOS-based  performance  measurement  pages  were  also  collected  and 
evaluated  at  various  points  during  the  simulated  mission.  The  resulting  process-performance 
relationships  were  accordingly  established  from  independent  data  sources. 

Robust  Team  Assessment  Measures 

The  study’s  measures  of  team  mission  performance  and  coordination  focused  on  behaviors 
that  were  collectible,  variable  across  crews,  and  operationally  relevant.  First,  we  considered  the 
constraints  of  the  training  environment  and  resources  available  to  determine  behaviors  that  could 
be  measured  reliably.  We  oriented  our  measurement  efforts  on  those  behaviors  that  we  could 
practically  expect  to  collect  on  a  weekly  basis. 

Second,  we  selected  team  behaviors  that  we  thought  were  most  likely  to  vary  across  crews. 
For  example,  many  training  scenarios  are  designed  so  all  crews  will  satisfy  the  overall  objectives 
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of  the  mission — successful  latinch,  receiving  fuel  during  the  AR,  meeting  the  time  on  target 
(TOT),  navigating  within  prescribed  accuracy  levels,  performing  the  designated  infils  and  exlBls, 
and  so  forth.  Sole  reliance  on  these  mission  outcome  measures  posed  problems  for  a  team  effec¬ 
tiveness  study  as  the  nonvariance  across  crews,  reflecting  in  essence  a  “ceiling  effect,”  could 
yield  negative  results  for  any  variable  of  interest.  Thus,  given  our  overarching  objective  of 
identifying  the  most  effective  MC-130P  aircrews,  we  attempted  to  emphasize  behaviors  that 
maximally  differentiate  strong  from  weak  crews.  Our  preliminary  testing  and  SME-interviews 
provided  invaluable  insights  regarding  high  payoff  areas  (e.g.,  the  five  T^RM  subprocesses)  and 
potential  behaviors  on  which  to  focus. 

Third,  the  observed  behaviors  we  selected  were  operationally  relevant.  Indeed,  operational 
realism  was  one  of  the  primary  considerations  in  selecting  the  five  T^RM  subprocesses  (TM,  SA, 
FA,  TE,  and  C3)  described  in  Table  1.  In  addition,  crewmembers  often  complain  about  the 
“soft”  topics  (group  cohesiveness,  leadership)  traditionally  taught  in  CRM  courses  and  their 
weak  connection  to  the  missions  crews  actually  fly.  The  T^RM  subprocesses  we  examined 
attempt  to  bring  crew  coordination  training  closer  to  the  CMT  environment,  including 
operationally  relevant,  behavioral  indices  of  team  coordination.  These  behaviors  may  then  be 
folded  back  into  training,  providing  crews  with  immediate  and  relevant  feedback. 

Multimeasure,  Multimethod  (MM-MM)  Mix  of  Variables 

The  study  of  team  coordination  and  mission  performance  is  clearly  a  multifaceted  problem. 
Hence,  it  is  not  surprising  that  a  MM-MM  mix  of  variables  is  required  to  achieve  a 
comprehensive,  systematic  investigation  of  the  topic.  As  used  here,  a  MM-MM  approach  refers 
to  employing  a  battery  of  objective  (e.g.,  computerized  timing  and  covmts)  and  subjective  (e.g., 
ratings)  measures  coupled  with  quantitative  and  qualitative  methods  of  data  analysis.  From  an 
experimental  perspective,  a  MM-MM  variable  mix  was  advisable  as  it  permitted  us  to  cast  a  wide 
net  in  order  to  tap  cognitive  processes  that  may  have  been  too  complex  had  we  used  a  single 
index.  Logistical!) .  too,  the  approach  has  appeal  as  it  is  fairly  robust  with  regard  to  potentially 
devastating  losses  of  partial  data  due  to  simulator  malfunctions  or  subject-crew  turbulence. 

Referring  back  to  Figure  1,  one  can  see  that  to  fully  explore  the  links  in  the  measurement 
model,  a  minimum  of  seven  measurement  instruments  is  needed.  Further,  even  if  a  researcher 
elects  to  focus  on  onh  select  portions  of  the  model  (e.g.,  process  and  performance  links),  a  MM- 
MM  mix  of  variables  allows  one  to  correlate  process  and  performance  measures  as  well  as 
provide  opportunities  to  assess  select  intercorrelations  among  different  types  of  objective  and 
subjective  performance  measures. 

In  the  present  research,  the  following  methods  and  measures  were  used: 

1.  An  SME  rated  and  observed  T^RM  processes  across  five  mission  phases.  This  was 
accomplished  by  using  headsets  to  monitor  mission  execution  and  from  over-the- 
shoulder  observations  during  MP. 
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2.  In  a  similar  fashion,  a  second  researcher  rated  and  observed  team  mission 
performance. 

3.  Instructors  rated  individual  crewmember  and  team  performance  across  each  mission 
phase. 

4.  Finally,  select  simulator  performance  pages  were  printed  out  as  each  mission  phase 
was  executed. 

Behaviorally  Anchored  Ratings  Scales  (BARS) 

BARS  are  formal  rating  instruments  that  contain  written  descriptions  of  the  behaviors 
associated  with  each  scale  value.  These  descriptions  function  as  referents  or  anchors,  and  aid 
evaluators  in  determining  the  quality  of  various  dimensions.  Referents  or  anchors  can  be  the 
simple  presence  or  absence  of  a  behavior  (e.g.,  prepared  a  mission  execution  checklist?  yes/no) 
or  the  quantitative  standards  that  must  be  met  (e.g.,  executed  drop  within  30  s  of  planned  drop 
time).  The  behavioral  anchors  serve  as  criterion  standards  by  which  the  evaluators  give  their 
ratings,  as  opposed  to  preference-based,  normative  comparisons  to  other  crews.  This 
standardization  is  designed  to  promote  reliability  of  ratings  across  evaluators  and  crews.  In  recent 
years,  the  BARS  methodology  has  become  more  commonly  used  in  crew  coordination  research. 
For  example,  the  US  Army  has  made  extensive  use  of  BARS  in  the  evaluation  of  their  aircrew 
coordination  course  (Grubb,  Leedom,  Simon,  &  Zeller,  1993;  Gmbb,  Simon,  Leedom,  &  Zeller 
1993). 

We  applied  a  BARS  approach  for  assessing  team  mission  p)erformance  and  the  quality  of 
MP  products.  After  extensive  SME  review,  analysis,  and  testing,  we  arrived  at  a  series  of  team 
performance  BARS  for  the  different  mission  phases  as  well  as  the  flight  charts  and  flight  plans 
created  during  MP  which  made  up  our  T-MPT.  On  the  process  side,  a  modified  BARS  approach 
was  used  for  the  T-MOT.  These  instruments,  as  well  as  the  other  instruments  used  in  our 
research,  are  described  in  the  following  subsection. 

Data  Collection  Instruments 

In  depicting  our  team  performance  measurement  model,  recall  that  Figure  1  displayed  a 
series  of  ovals  that  referred  to  the  instruments  we  used  during  the  flow  of  training  activities.  The 
following  paragraphs  describe  the  purpose  and  scope  of  each  instrument  used  in  this  research 
effort,  along  with  pertinent  examples  to  illustrate  the  information  items  that  form  its  content. 

Crewmember  Background  Survey  (CBS) 

The  CBS  is  a  self-report  tool  that  captures  relevant  background  information  from  each 
crewmember.  The  instrument's  purpose  is  to  enable  us  to  build  a  descriptive  profile  of  each 
crewmember's  flight,  operational  mission,  weapons  system,  and  organizational  experience. 
Besides  requesting  total  flight  experience  information,  the  CBS  asks  for  each  crewmember’s 
estimate  of  recent  flight  experience  with  the  other  crewmembers  attending  ART.  Aggregated 
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across  each  crew,  the  CBS  was  used  to  create  descriptive  profiles  at  the  aircrew-level  (see 
Appendix  A).  This  instrument  was  administered  on  the  first  day  of  ART,  immediately  upon 
crewmember  completion  of  course  registration  and  prior  to  the  beginning  of  classroom  training 

Team-Mission  Observation  Tool  (T-MOT). 

The  T-MOT  was  an  integral  part  of  our  total  assessment  strategy.  This  instrument  was 
designed  to  aid  in  recording  specific  individual  and  team  coordination  behaviors  that  fell  into 
distinct  T^RM  subprocesses,  and  which  occur  during  discrete  mission  phases.  Measurement  was 
accomplished  using  5-point  Likert  scales  and  SME  observations  of  critical  coordination  behav¬ 
iors  tied  to  a  complex  CMT  scenario. 

The  T-MOT  supported  recording  and  analyzing  both  individual  crewmember  and  aircrew 
team  behaviors  within  the  five  key  T^RM  subprocesses  (TM,  FA,  etc.)  across  critical  mission 
phases.  The  T-MOT  structures  an  SME’s  first-hand  observations  of  complex  task  performance 
during  both  MP  and  mission  execution.  An  internally  consistent  and  reliable  “record  by 
exception”  measurement  philosophy  was  employed  for  capturing  instances  of  extreme  crew 
coordination  behaviors  demonstrated  during  CMT.  Once  collected,  content  analyses  were 
performed  on  the  recorded  behaviors  to  permit  comparisons  of  fi:equency,  quality,  and/or 
intensity  across  teams.  This  qualitative  analysis  supplemented  quantitative  analyses  performed 
on  the  rating  data  that  were  also  produced  with  this  instrument. 

The  primary  data  collection  method  used  in  the  T-MOT  was  over-the-shoulder,  structured 
observations,  a  method  sanctioned  by  Air  Force  Instruction  36-2243  (AFI,  1994).  This  technique 
has  been  successfully  used  in  previous  MP  and  MR  studies  (Spiker  &  Nullmeyer,  1995a;  1995b). 
While  there  was  a  “script”  to  help  our  SME  structure  his  observations,  he  was  also  free  to  record 
“by  exception”  activities  in  which  attitudes,  behaviors,  and  cognitions  that  seemed  unusually 
strong  or  weak  were  noted.  This  technique  was  principally  used  in  observing  T^RM  subprocesses 
by  our  highly  experienced  SME.  When  performed  by  a  trained  SME,  direct  observations  achieve 
respectable  levels  of  reliability  (Tourville,  Spiker,  Silverman,  &  Nullmeyer,  1996). 

The  T-MOT  is  divided  into  subsections  devoted  to  a  particular  T^RM  subprocess  within  a 
particular  mission  phase.  Specific  YES/NO  checklist  items  are  provided  within  each  subsection 
and  assessed  by  the  SME.  Items  include  an  explanation  field  available  to  record  notable 
behaviors.  Figure  2  displays  an  example  item  fi-om  the  TM  subprocess.  Appendix  B  contains 
example  items  from  the  TE,  FA,  SA,  and  C3  subprocesses. 

Additionally,  we  used  a  5-point  rating  scale  (1  =  lowest  to  5  =  highest)  to  provide  quan¬ 
titative  assessments  across  T^RM  subprocesses,  crewmember  positions,  and  mission  phases.  This 
technique  proved  to  be  a  fairly  efficient  way  to  generate  a  large  amount  of  data  within  a  well- 
defined  structure  in  which  the  observer  used  rules  to  assign  ratings  to  specific  attributes  of 
T'RM. 


29 


Time  Management  (TM);  Involves  the  ability  of  the  combat  mission  team  to  employ  and  manage  limited  time 
resources,  so  that  all  tasks  receive  sufficient  time  to  be  performed  correctly,  and  critical  tasks  arc  not  omitted. 


1 .0  An  end-mission  planning  time  should  be  indicated  up  front  -  most  likely  by  an  emergent  “  leader.” 

a.  Did  any  crewmember  indicate  the  need  for  an  end-mission  planning  time? .  YES/NO 

(Explain) _ _ _ 

b.  Was  that  time  noted  by  all  other  crewmembers? .  YES/NO 

(Explain) _ _ _ 

c.  Did  any  crewmember  designate  activities  to  establish  a  proper  balance  between  their  own  authority,  time 

available,  and  crewmember  participation? .  YES/NO 

(Explain) _ _ _ _ 

d.  Was  adequate  mission  preparation  time  allocated  for  a  comprehensive  pre-mission  briefing? .  YES/NO 

(Explain)  _ _ _ _ 


Figure  2.  Example  TM  Item  from  the  T-MOT. 

The  T-MOT  included  brief  descriptions  of  each  mission  phase  (see  Figures  3-7)  and  each 
phase  was  assessed  separately  during  Day  5  of  ART.  The  phases  were  assessed  in  their  naturally 
occurring  sequence,  as  a  series  of  mission  events  that  occur  up  to,  during,  and  immediately  after 
the  particular  mission  objective. 


As  shown  in  Figure  3,  each  crewmember’s  demonstrated  behavior  during  MP  was  individu¬ 
ally  rated  by  our  SME-observer,  using  the  1  to  5  scale,  across  all  five  T^RM  subprocesses.  A 
similar  format  was  used  to  rate  the  other  mission  phases. 


Mission  preparation  (MP)  procedures:  The  objective  is  to  conduct  mission  planning  and  briefing  activities  that 
allow  sufficient  preparation  of  a  comprehensive  mission  execution  plan.  This  plan  will  be  prepared  with  considerations 
for  a  medium  threat  environment,  all  major  mission  events  and  activities;  and  mission  operations  procedural  constraints. 


AC 

CP 

Nav  1 

Nav  2 

FE 

CSO 

1. 

Situation  Awareness 

2. 

Function  Allocation 

3. 

Tactics  Employment 

4. 

Time  Management 

5. 

Command,  Control,  &  Comm. 

Figure  3.  Matrix  of  Ratings  Used  in  the  MP  Segment  of  the  T-MOT. 


Figure  4  illustrates  one  of  the  measurement  items  from  the  T-MOT.  This  item  was  used  to 
record  the  demonstration  of  C3  behaviors  during  the  LL  phase.  As  with  the  other  items,  the 
observer  first  indicated  whether  the  behavior  was  present  or  absent;  he  then  provided  explanatory 
comments  in  the  space  available. 
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Low-Level  (LL)  tactical  operations  procedures:  The  objective  is  to  conduct  NVG  low-level  flight  en  route 
to  specific  mission  events  using  proper  tactical  mission  management  procedures  (altitude,  airspeed,  terrain 
masking,  etc.)  for  a  medium-threat  environment. 


2.0  (C3)  CSO  receives  incoming  message  that  (one)  helicopter  has  ditched.  The  crew  should  spend  <5  min  dealing 
with  the  problem  (including  time  for  CSO  to  filter  info).  There  need  not  be  an  excessive  amount  of  discussion  about 


the  problem’s  solution. 

a.  Was  this  event  handled  by  one  focal  crewmember  (versus  a  full  crew  emphasis)? .  YES/NO 

(Explain)  _ _ _ 

b.  Did  the  CSO  filter  the  message  appropriately  ? .  YES/NO 

(Explain) _ 

c.  Were  reasonable  options  presented  for  dealing  with  the  message? .  YES/NO 

(Explain) _ _ _ _ 

d.  Was  an  appropriate  decision  (outcome)  ultimately  concluded? .  YES/NO 

(Explain) _ _ _ 


Figure  4.  Example  of  a  C3  Item  from  the  LL  Segment  of  the  T-MOT. 


Figure  5  illustrates  another  measurement  item  from  the  T-MOT.  This  item  was  used  to 
record  the  demonstration  of  TE  behaviors  during  the  AR  phase. 

Air  refueling  (AR)  procedures:  The  objective  is  to  successfully  conduct  tactical  in-flight  AR  of  (multiple) 

MH-53 J  Pave  Low  helicopters  within  prescribed  time,  course,  and  altitude  in  a  medium-threat  environment. 

3.0  (TE)  The  AR  should  be  completed  early,  so  they  can  escape  hostile  airspace  quicker.  This  also  gives  the 
crew  additional  flex  time  for  later  in  the  mission,  when  mission  events  get  tight  around  LZ  #2. 


a.  Did  the  crew  exercise  proper  tactical  refueling  (phase)  management  procedures? .  YES/NO 

(Explain) _ 

b.  ARCP  AT  A _ -  Acceptable? .  YES/NO 

(Explain) _ _ _ 

c.  EAR  Time  _ _  -  Acceptable? .  YES/NO 

(Explain) _ _ _ 


Figure  5.  Example  of  a  TE  Item  from  the  AR  Segment  of  the  T-MOT. 


Figure  6  depicts  a  TE  measurement  item  from  the  AD  phase. 

Airdrop  (AD)  procedures:  The  objective  is  to  successfully  conduct  CARP  airland  of  SF  personnel  within 
prescribed  time,  course,  and  altitude  constraints  in  a  medium-threat  environment. 

4.0  (TE)  Technical  proficiency  of  airdrop  should  be  rated  by  exception. 

a.  Were  there  any  problems  noted  during  the  airdrop  procedure? .  YES/NO 

(Explain) _ 


Figure  6.  Example  of  a  TE  Item  from  the  AD  Segment  of  the  T-MOT. 


31 


Figure  7  presents  another  TE  measurement  item.  This  one  is  from  the  I/E  phase. 


Infil/Exfil  (I/E)  procedures:  The  objective  is  to  successfully  conduct  covert  infll  and/or  exfil  at  multiple  tactical 
landing  sites  for  transload  purposes  within  prescribed  time,  course,  and  altitude  constraints  in  a  medium-threat  environment. 


5.0  (TE)  Crews  can  have  serious  problems  making  the  approach  due  to  poor  visibility  and  NVG  conditions. 

a.  Did  the  crew  have  problems  with  the  approach? .  YES/NO 

(Explain)  _ _ 


Figure  7.  Example  of  a  TE  Item  from  the  I/E  Segment  of  the  T-MOT. 


Lastly,  the  T-MOT  included  a  summary  section  (see  Figure  8)  where  each  crewmember  and 
crew  as  a  whole  were  rated  on  the  five  T^RM  subprocesses  by  each  mission  phase.  This  was 
followed  by  a  summary  table  where  overall  crew  coordination  for  each  crewmember  and  crew 
was  assessed  (bottom  section  of  Figure  8). 


OVERALL:  Summarize  each  crewmember’s,  and  the  crew’s,  demonstrated  overall  CRM  behaviors,  during  each 
mission  phase  as  indicated,  from  1  to  5  (Use  rating  scale). 


OVERALL  (CRM)  Rating 

AC 

CP 

LN 

RN 

FE 

CSO 

C^W 

1. 

Mission  Preparation  (MP) 

2. 

Low-Level  (LL) 

3. 

Air  Refueling  (AR) 

4. 

Airdrop  (AD) 

5. 

InfiiyExfil  (I/E) 

OVERALL  (CRM)  Ratine 

CP 

LN 

RN 

FE 

CSO 

FINAL  ASSESSMENT 

Figure  8.  T-MOT  Overall  Crew  Coordination  Assessment  Section. 


Team-Mission  Performance  Tool  (T-MPT) 


The  T-MPT  was  designed  to  aid  in  evaluating  the  mission  performance  that  was  demon¬ 
strated  during  the  tactical  MP  and  execution  phases.  This  instrument  provided  a  structured 
method  for  the  second  researcher  to  rate  the  quality  of  individtial-  and  team-generated  mission 
products  developed  during  the  MP  phase,  as  well  as  to  provide  anchored  ratings  of  demonstrated 
performance  across  the  mission  execution  phases.  The  quality  of  the  mission  products  developed 
by  the  aircrew  team  or  individual  crewmembers  is  one  index  of  team  mission  pierformance. 

Figure  9  depicts  a  BARS  item  from  the  T-MPT  that  was  used  to  rate  mission  flight  charts 
developed  during  the  MP  phase.  The  scale  was  used  to  assess  the  quality  of  the  flight  charts 
generated  by  the  LN,  RN,  and  the  pilots. 
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1 

2 

3 

4 

5 

“  Poor. 

-  Marginal. 

-  Adequate. 

-  Outstanding. 

-  Exceptional. 

-  Incomplete  data. 

-  Insufficient  or 

-  Threats  plotted. 

-  Threat  rings 

-  Threat  contour 

- 

inaccurate  docu- 

plotted. 

shading 

-  General  lack  of 
documentation. 

mentation. 

-  Unaccounted  for 

-  Most  threat  rings 
plotted. 

-  Deviation  plan 
clearly  drawn 

provided. 

-  Deviation  plan 

-  General  quality 

discrepancies 

-  Deviation  plan 

and  visible  for 

and  threat  infor- 

of  preparation  is 
poor. 

between  LN, 

RN,  and  CP 

clearly  drawn. 

NVG  conditions. 

mation  high¬ 
lighted  for  NVG 

charts. 

-  Appropriate  alti¬ 
tude  considera¬ 

-  Appropriate  alti¬ 
tude  and  terrain 

conditions. 

-  Deviation  plan 
minimally 

tions  made. 

considerations 
made  and 

-  Documentation 
in  excess  of 

prepared. 

-  Required 
checklist 
annotations 

explicitly 
represented  in 
the  deviation 

minimum 

requirements. 

-  Marginal 
quality. 

made. 

plan. 

-  Threat  labels. 

Score 

Explain 

LN  Mission  Chart 

RN  Mission  Chart 

Pilot  Mission  Chart 

Figure  9.  Example  BARS  Item  from  the  MP  Portion  of  the  T-MPT. 


Figure  10  presents  a  BARS  item  from  the  T-MPT  that  was  used  to  score  performance 
during  the  AR  phase.  BARS  were  also  used  to  assess  LL,  AD,  and  I/E  performance.  As  with  the 
planning  products  described  above,  the  proficiency  of  mission  phase  behavior  demonstrated  by 
the  aircrew  team  is  another  index  of  team  mission  performance. 

Instructor  Rating  Instrument  (IRI) 

The  IRI  was  designed  to  capture  the  unique  perspectives  of  training  instructors  as  they 
rated  the  demonstrated  performance  of  their  own  trainees  in  the  context  of  overall  crew 
performance.  For  the  MC-130P  WST,  separate  instructors  train  the  pilots  (AC  and  CP), 
navigators  (LN  and  RN),  CSO,  and  FE.  Each  crewmember  was  assessed  by  their  instructors 
using  a  YES/NO  checklist  and  a  1-5  Likert  scale.  These  items  and  ratings  covered  issues  of 
demonstrated  MP  and  mission  execution  performance  that  were  relevant  to  that  particular 
trainee’s  crew  position.  In  other  words,  each  IRI  was  specifically  designed  and  tailored  to  assess 
the  particular  instructor  s  student-crew  position  roles  and  responsibilities  throughout  the  mission. 

Secondarily,  the  IRI  was  designed  to  capture  the  instructor’s  unique  subject-matter 
expertise  and  perspective  by  asking  for  separate  ratings  of  their  individual  students,  as  well  as  the 
aircrew  team  as  a  whole,  across  mission  phases.  In  this  manner,  another  index  of  team  mission 
performance  was  provided  that  was  independent  of  the  assessment  provided  by  the  researchers 
armed  with  the  T-MPT  and  T-MOT. 


33 


AIR  REFUELING 


1 

2 

3 

4 

5 

-  Poor  time  control 

to  ARCP.  Arrives 
at  ARCP  earlier 
than  I-minute 

prior  to  planned 
ARCT. 

-  Poor  time  control 
to  ARCP.  Arrives 
at  ARCP  greater 
than  +  1  minute 
(or  more)  late  to 
ARCP 

•  Minimal  time 

control  to  ARCP. 

-  Arrives  at  ARCP 
in  window  of  from 

1 -minute  to  1- 
second  earlier  than 
planned  ARCT. 

-  Unexplained 
maneuvering  off 
refueling  track. 

-  Adequate  time 

control  to  ARCP. 

-  Arrives  at  ARCP 
in  window  of  from 
30-seconds  to  60- 
seconds  later  than 
planned  ARCT. 

-  Maintains 
refueling  track. 

-  Outstanding  time 
control  to  ARCP. 

-  Arrives  at  ARCP 
in  window  of  from 
15-seconds  to  30- 
seconds  later  than 
planned  ARCT. 

-  Exceptional  time 
control  to  ARCP. 

-  Arrives  at  ARCP 
in  window  of  from 
on-time  to  15- 
seconds  later  than 
planned  ARCT. 

Was  the  AR  operation  successful? .  YES/NO 

Explain  _ _ 


Figure  10.  Example  BARS  Item  from  the  AR  Portion  of  the  T-MPT. 
Crewmember  Self-Assessments  (as  part  of  the  TMAQ2) 

In  addition  to  post-ART  CRM  attitude  assessments  (analyzed  in  Spiker,  Silverman, 
Tourville,  &  Nullmeyer,  in  press)  the  TMAQ2  included  a  section  that  had  questions  regarding 
the  crewmember’s  perceptions  of  his  crew’s  mission  readiness,  mission  confidence, 
coordination,  and  performance.  This  instrument  was  administered  immediately  upon  the 
student’s  completion  of  his/her  ART  requirements. 

Procedure 

The  instruments  described  above  reflect  the  four  primary  sources  of  data  for  this  research 
effort:  (a)  student  questionnaires,  (b)  instructor  questionnaires,  (c)  structured  researcher 
observations,  and  (d)  lOS  page  printouts.  Procedural  steps  and  schedule  for  collecting  data  from 
each  of  these  sources  are  detailed  below. 

Student  Questionnaires 

Following  the  aircrews  registration  for  ART,  researchers  administered  the  CBS  and  the 
TN^Ql.  During  any  given  week  of  training,  various  crews  in  addition  to  MC-130P  crews 
arrived  for  ART.  The  CBS  and  TMAQl  were  administered  to  all  of  the  crews  who  showed  up  en 
masse,  which  were  sometimes  as  many  as  40  crewmembers.  Prior  to  administering  the 
questionnaire,  one  of  the  researchers  briefly  described  the  research  project  and  focus,  the  purpose 
of  the  information,  and  the  researchers’  roles  in  the  upcoming  days  of  ART.  The  researcher  then 
distributed  the  questionnaires  and  remained  available  for  questions  or  explanation  until  the 
trainees  completed  the  CBS  and  TMAQl.  The  entire  process  took  approximately  30  minutes, 
which  had  been  formally  scheduled  into  the  ART  curriculum. 
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Following  the  simulator  mission  (their  last  training  event)  on  Day  5  of  ART,  the 
researchers  administered  the  TMAQ2,  which  included  the  identical  attitude  questions  from  the 
TMAQl  with  some  additional  self-assessment  questions  regarding  their  individual  and  crew 
coordination  and  performance  for  that  dayis  mission.  The  TMAQ2  was  usually  completed  prior 
to  instructor/crew  debriefs  of  the  mission.  The  total  time  for  the  students  to  complete  the 
TMAQ2  ranged  from  15  to  25  minutes. 

Instructor  Questionnaires 

MP  and  mission  execution  performance  data  were  collected  from  instructors  using  the  IRI. 
The  IRIs  were  distributed  to  each  instructor  at  the  beginning  of  the  MP  session  for  the  Day  5 
mission.  As  mentioned,  each  IRI  was  customized  with  crew  position-specific  checklist  items  and 
phase-specific  performance  ratings.  The  IRI  also  included  two  overall  crew  performance  and 
coordination  questions.  Instructors  usually  filled  out  the  questionnaires  in  two  stages — 
completing  the  MP  performance  questions  immediately  following  the  planning  session  and  the 
mission  execution  section  following  the  simulator  mission  (e.g.,  while  the  students  completed  the 
TMAQ2).  Instructors  took  approximately  10  minutes  (total)  to  complete  the  IRIs. 

Early  on  in  the  data  collection  process,  we  realized  that  the  instructors  were  often  “double- 
booked,”  i.e.,  had  other  duties  assigned  concurrently  with  their  training  duties.  This  meant  that 
some  instructor  crew  positions  were  not  present  during  planning.  When  this  happened,  these 
instructors  would  only  complete  the  mission  execution  portion  of  the  IRI.  This  did  not  impede 
our  research  efforts,  though,  as  our  main  source  of  performance  data  came  from  the  researcher- 
structured  observations  discussed  below. 

Researcher-Structured  Observations 

Researcher  observations  were  the  primary  source  of  process  and  performance  data,  with 
one  researcher  responsible  for  collecting  process  data  and  the  other  responsible  for  performance 
data.  The  process  data  were  recorded  on  the  T-MOT  by  a  highly  trained,  former  MC-130P 
navigator.  During  MP,  this  individual  served  as  a  participant-observer  (P/0).  This  relationship 
was  deemed  appropriate  as  our  SME  possessed  a  comprehensive  understanding  of  current  SOF 
doctrine  and  extensive  flight,  training,  and  simulation  experience  in  SOF  operations  through 
training  exercises  and  operational  missions.  In  addition,  the  P/0  approach  has  been  successfully 
used  in  other  team  training  research  realms  (e.g.,  Oser,  et  al.,  1996).  The  P/O’s  primary  responsi¬ 
bility  was  to  perform  data  collection  tasks  (observer)  using  the  T-MOT.  Secondarily,  he  served 
as  the  senior  controller  (participant)  for  such  items  as  professional  presentation  of  the  training 
materials,  development  of  the  scenario  script,  role-playing  as  liaison  officer  with  outside 
(simulated)  agencies,  and  providing  mission  debrief  support.  The  simulator  training  instructors 
collaborated  with  our  P/0,  and  were  also  responsible  for  either  controlling  the  computer 
simulation  and  role-playing  additional  higher  C&C  functions. 

During  mission  execution,  the  SME-researcher  observed  and  monitored  crews  from  an 
intercom  station  located  outside  the  MC-130P  WST.  The  intercom  station  was  situated  in  front  of 
four  instructor-operator  screens  which  repeated  the  instructor  inputs  from  inside  the  MC-130P 
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WST.  From  this  vantage  point,  the  SME-researcher  was  able  to  make  informed  observations  and 
assessments  based  on  personal  knowledge  and  expertise  concerning  the  mission,  coordination 
performance,  and  monitoring  instructor  inputs. 

The  T-MOT’s  items  were  organized  by  mission  phase  and  followed  the  prescribed 
sequence  of  mission  events.  Overall  ratings  and  summary  assessments  were  filled  in  immediately 
following  the  simulator  mission.  The  completed  T-MOT  was  reviewed  early  the  following  week 
to  ensure  that  no  assessment  items  were  omitted.  On  rare  occasions  during  this  review,  a  change 
or  two  would  be  made  to  an  overall  assessment  due  to  recognition  of  a  transcription  error  or 
something  notable  having  been  overlooked. 

Performance  data  were  collected  similarly.  However,  the  researcher  who  collected  perform¬ 
ance  data  did  not  serve  as  a  P/0  during  MP.  She  merely  observed  crew  performance  keeping  in 
mind  the  criteria  established  in  the  T-MPT.  Her  notes  and  observations  captured  such  items  as: 
the  number  of  briefings  each  crew  gave,  the  contents  of  the  briefings,  who  performed  the 
briefings,  the  number  of  charts  created,  etc.  All  of  these  notes  facilitated  post-mission  completion 
of  the  T-MPT.  Similar  types  of  notes  were  taken  during  mission  execution  (e.g.,  time  of  AD, 
time  to  complete  AR).  The  researcher  was  located  in  the  SNS  collocated  with  the  Instructor  CSO 
and  CSO  while  the  mission  transpired.  From  the  SNS,  she  monitored  the  crew  communications, 
flight  path,  and  threat  laydown.  It  was  also  fi-om  this  location  that  various  lOS  pages  were 
selected  and  printed  out  (see  next  subsection),  to  aid  in  later  completion  of  the  T-MPT. 

At  the  conclusion  of  the  simulator  mission,  the  researcher  collected  all  products  (flight 
plans,  charts,  SCAs,  execution  checklists,  etc.)  that  the  crew  created  for  the  mission.  These 
materials  were  then  used  to  complete  the  BARS  items  on  the  T-MPT. 

This  combination  of  materials — ^notes,  lOS  pages,  and  crew  products — ^was  collectively 
evaluated  using  the  T-MPT.  For  example,  AD  performance  was  evaluated  using  the  BARS  for 
AD  where  the  actual  performance  was  checked  using  the  lOS  ground  track  map  (GTM)  page,  the 
prepared  chart,  and  the  recorded  drop  time.  Besides  rating  the  mission  phase,  the  core  products 
(flight  plans  and  flight  charts)  were  rated  using  a  BARS.  The  quality  and  quantity  of  additional 
products  and  briefings  were  also  evaluated  using  the  T-MPT.  Due  to  the  classified  nature  of  the 
mission  and  materials,  the  products  were  destroyed  and  the  charts  were  erased  once  assessments 
were  completed. 

lOS  Pages 

As  mentioned  above,  the  primary  purpose  of  printing  out  selected  lOS  pages  was  to  aid  the 
researcher  in  assigning  the  behaviorally  anchored  performance  ratings  (e.g.,  compare  planned 
ground  track  to  flight  path  actually  flown)  to  each  crew.  The  following  lOS  pages  were  printed 
out  during  the  mission:  (a)  GTM  pages  at  each  waypoint;  (b)  AR  pages  at  initial  point,  control 
point,  and  at  least  two  during  the  contact  phase;  and  (c)  GTM  pages  during  approach  and  while 
landing  for  the  I/E.  The  GTM  pages  for  the  AD,  AR,  and  I/E  waypoints  were  the  most  critical  for 
completing  the  T-MPT. 
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RESULTS 


T^RM  Process  and  Team  Mission  Performance 
Rating  Data  Structure 

Table  9  depicts  the  data  structure  for  the  process  and  performance  ratings.  Recall  that  all 
process  ratings  were  obtained  from  the  T-MOT  whereas  the  performance  data  came  from  the  T- 
MPT,  in  which  separate  raters  were  used  for  the  two  instruments  in  all  cases.  An  overall  process 
rating  and  specific  process  ratings  for  SA,  TE,  TM,  FA,  and  C3  were  generated  for  the  whole 
mission  and  the  five  mission  phases  (MP,  LL,  AD,  AR,  and  I/E),  resulting  in  the  6  x  6  matrix  of 
process  ratings  depicted  at  the  top  of  Table  9.  Mission  performance  ratings  were  provided  for 
the  five  mission  phases,  and  a  total  mission  performance  rating  was  generated  by  calculating  the 
sum  of  the  five  phase-specific  ratings  for  each  crew,  a  value  that  we  will  refer  to  as  PerfSum, 
resulting  in  the  six-element  array  of  performance  ratings,  depicted  at  the  bottom  of  Table  9.  With 
respect  to  PerfSum,  we  used  a  sum  of  the  five  mission-phase  performance  ratings  because  the 
skills  required  in  each  phase  are  so  different  that  we  did  not  believe  that  an  overall  rating  would 
be  reliable  or  valid.  The  PerfSum  variable  was,  itself,  multivariate  in  nature  since  it  was  based  on 
a  composite  of  MP  performance  (as  determined  by  ratings  and  tallies  of  key  mission  planning 
products,  the  details  of  which  are  described  later)  and  the  overall  performance  ratings  obtained 
for  each  phase  of  the  mission  (LL,  AD,  etc.). 


Table  9.  Data  Structure  for  Ratings  of  T^RM  Subprocesses  and  Combat  Mission  Perfonnance 


Process 

Ratings 

Whole 

Mission 

Mission 

Preparation 

Low 

Level 

Air 

Drop 

Air 

Refueling 

Infil/Exfil 

Overall 

Process 

SA 

TE 

TM 

FA 

C3 

Performance 

Ratings 

(PerfSum) 

Phase- 

specific  rating 

Phase- 

specific  rating 

Phase- 

specific  rating 

Phase- 

specific  rating 

Phase- 

specific  rating 

Process  and  Performance  Rating  Correlations 

The  majority  of  analyses  that  we  report  in  this  subsection  involve  Pearson  product  moment 
correlations  between  various  pairs  of  crew-level  rating  data  associated  with  T^RM  processes  and 
combat  mission  performance.  These  correlations  are  summarized  in  Table  10. 
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Before  discussing  the  results,  we  must  first  describe  some  conventions  for  interpreting  the 
contents  of  this  rather  complex  summary  table.  We  use  the  first  column  to  describe  the  types  of 
analyses  that  are  contained  in  the  subsequent  cells  of  that  row.  Column  2  of  the  table  refers  to 
correlations  involving  pairs  of  performance  and  process  ratings  reflecting  the  whole  mission  and 
the  remaining  five  columns  reflect  ratings  of  behaviors  performed  within  specific  mission 
phases. 

The  rows  in  the  table  correspond  to  categories  of  correlations  that  cross  mission  elements. 
The  top  row  depicts  correlations  between  our  most  global  process  rating  (the  overall  process 
rating  for  the  entire  mission)  and  the  six  performance  ratings.  The  second  row  summarizes 
correlations  between  our  most  global  performance  rating,  PerfSum,  and  the  overall  process 
ratings  generated  for  each  mission  phase.  The  third  row  summarizes  correlations  between  phase- 
specific  overall  process  ratings  and  phase-specific  performance  ratings.  The  remaining  five  rows 
reflect  correlations  between  performance  ratings  and  each  of  the  five  subprocess  ratings  within 
each  of  the  six  mission  categories.  It  should  be  noted  that  the  first  column  contains  correlations 
between  PerfSum,  which,  as  mentioned,  was  computed  as  a  sum  of  the  five  mission  element 
ratings,  and  a  summary  rating  for  each  subprocess  that  takes  all  mission  phases  into  account. 

Table  10.  Correlations  Among  Ratings  of  T^RM  Processes  and  Combat  Mission  Performance 


Process/Performance 

Whole 

Mission 

MP 

LL 

AD 

AR 

I/E 

Overall  Process  x 
Performance 

.86* 

(.89)* 

.78^ 

.74* 

.39 

.52 

.41 

Phase-Specific  Process  x 
Performance  Sum 

.78* 

.57 

.79* 

.75* 

.77* 

Phase-Specific  Process  x 
Phase-Specific  Performance 

.68 

.46 

.54 

.70 

.67 

SA  Subprocess  x 
Performance 

.76* 

.48 

.47 

.59 

.28 

.65 

TE  Subprocess  x 
Performance 

.78* 

.27 

.06 

.54 

.81* 

.55 

TM  Subprocess  x 
Performance 

.83* 

.41 

.36 

.66 

.51 

.64 

FA  Subprocess  x 
Performance 

.75* 

.22 

.61 

.55 

.55 

.60 

C3  Subprocess  x 
Performance 

.08 

.14 

.09 

.32 

.37 

.30 

Experiment-wise  and  nominal  criterion  levels  for  significance  testing 
*Pew<  05,  Pnom<  001  Bonferroni  adjustment  assuming  46  tests;  critical  r  =  .74 


Within  this  structure,  the  most  basic  question  we  ask  is  whether  there  is  a  statistically 
significant  relationship  between  mission  performance  (PerfSum)  and  overall  crew  T^RM  across 
the  entire  mission.  The  first  row,  first  column  depicts  the  correlations  between  the  overall  rating 
of  crew  coordination  for  the  whole  mission  and  the  rating  of  crew  performance  for  the  total 
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mission  (PerfSum).  Outlined  in  bold,  this  whole-mission  T^RM  by  PerfSum  correlation  is  the 
single  most  important  analysis  in  the  table,  as  it  provides  our  most  global,  and  hopefully  most 
robust,  assessment  of  the  T^RM  process-CMT  performance  relationship. 

To  further  explore  the  relationships  between  process  and  performance,  we  looked  within 
the  five  primary  mission  phases.  Our  data  structure  allowed  three  ways  to  view  phase-specific 
process  and  performance  relationships:  expand  performance  ratings  to  reflect  individual  mission 
phases  and  correlate  with  the  whole  mission  T^RM  rating  (Row  1),  expand  process  ratings  and 
correlate  with  PerfSum  (Row  2),  or  expand  both  (Row  3). 

The  five  entries  on  the  right-hand  side  of  the  top  row  depict  correlations  between  the 
whole-mission,  overall  process  rating,  and  the  five  performance  ratings  for  specific  mission 

elements.  For  this  row,  the  five  column  headings  which  contain  the  analytic  results _ MP,  LL 

-^^5  and  I/E  ^reflect  the  origin  of  the  performance  ratings. 


The  second  row  in  Table  10  depicts  the  correlations  between  phase-specific  overall  process 
ratmgs  and  PerfSum.  The  five  column  headings  which  contain  the  analytic  results  now  refer  to 
toe  phase  m  which  toe  overall  process  measure  was  taken,  not  performance.  The  first  data  cell  in 
toat  row  has  been  shaded  out  since  its  combination  is  identical  to  toe  corresponding  cell  in  toe 
first  row  (i.e.,  overall  process  and  overall  performance). 


pie  third  row  in  Table  10  follows  a  similar  logic  to  toat  in  toe  first  and  second  rows.  This 
time,  however,  the  values  refer  to  correlations  between  phase-specific  process  ratings  and  phase- 
specific  ratings  of  mission  performance.  Thus,  toe  five  mission  phase  column  headings  now 
contain  ratmgs  m  which  both  process  and  performance  are  phase-specific. 

The  correlations  in  toe  lower  part  of  toe  table  explore  toe  impact  of  T^RM  process  further 
by  considenng  toe  relationships  between  toe  various  T^RM  subprocesses  and  toe  corresponding 
performance  rating.  TTie  five  lightly  shaded  cells  in  toe  first  column  depict  toe  correlations 
between  each  of  toe  overall  subprocess  ratings  (i.e.,  assessed  across  mission  elements)  with  total 
mission  performance,  or  PerfSum.  The  remaining  25  cells  contain  toe  most  detailed  view  of  toe 
process-performance  relationship.  They  depict  toe  correlations  between  each  subprocess— within 
each  mission  phase— with  toe  corresponding  performance  rating  for  toat  mission  phase. 

Statistical  Testing  Considerations 


Because  all  tests  reported  in  this  section  use  crew  as  toe  unit  of  analysis,  our  total  N  of  1 1 
and  toe  resultmg  9-degrees-of-ffeedom  (i.e.,  for  t-tests,  df  =  N-2  =  9)  seem  rather  small  to 
achieve  the  statistical  power  toat  one  would  like  to  establish  a  strong  process-performance 
relationstop.  However,  toe  11  crews  in  our  sample  actually  constitute  a  substantial  percentage 
(26)  of  toe  population  of  approximately  42  SOF  MC-130P  aircrews  toat  go  through  ART  ea?h 
year.  Consequently,  because  we  sampled  a  sizable  proportion  of  toe  population,  we  are  able  to 
redixe  om  esbmated  vmance  of  toe  sample  mean  by  using  a  finite-population  correction 
coefficient  (Winkler  &  Hays,  1975).  The  correction  coefficient  decreases  toe  observed  sample 
vanance  by  the  square  root  of  (N-l)/N-n,  where  N  is  toe  population  size  and  n  is  toe  sample  sL 
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In  our  case,  the  reported  t-values  below  have  been  increased  by  20%,  reflecting  a  1.2  finite- 
population  coefficient  multiplier. 

Due  to  the  exploratory  nature  of  many  of  our  research  questions,  and  the  need  to  perform  a 
large  number  of  statistical  tests,  we  used  a  Bonferroni  adjustment  as  a  way  to  keep  our  overall,  or 
experimentwise,  alpha  level  from  exceeding  the  desired  (nominal)  level.  The  Bonferroni 
technique  is  a  conservative,  though  effective,  way  to  avoid  inflating  the  alpha  level  (and  hence 
the  likelihood  of  a  Type  I  error)  by  “snooping”  through  one’s  data  to  locate  the  largest  effect 
(Harris,  1994).  The  adjustment  is  made  by  dividing  the  desired  experimentwise  alpha  level  by  the 
number  of  tests  that  are  performed  in  a  given  cycle  of  testing. 

As  shown  at  the  bottom  of  Table  10,  we  have  employed  a  conservative  nominal  alpha  level 
of  .001  to  control  for  the  fact  that  we  are  performing  46  statistical  tests.  Since  all  correlations  are 
tested  against  a  null  hypothesis  of  0,  this  corresponds  to  a  critical  t-value  of  4.19  (df  =  9,  one- 
tailed).  Given  the  finite  population  correction  described  above,  this  means  that  our  reported 
correlations  have  to  reach  a  criterion  of  at  least  .74  to  be  significant. 

Primary  Result  -  Overall  T^RM  Process  and  Overall  Mission  Performance 

Turning  to  the  correlations  in  Table  10,  we  find  that  our  primary  hypothesis — a  strong, 
positive  relationship  between  overall  T^RM  process  and  overall  mission  performance — is 
supported  by  the  data.  This  is  evident  from  the  large  correlation  coefficient,  .86  (t  =  6.143,  df  =  9, 
p  <  .05,  two-tailed),  that  appears  in  the  upper  left-hand  cell  of  the  table. 

To  appreciate  the  highly  linear  relationship  between  T^RM  process  and  mission 
performance,  Figure  1 1  depicts  the  scatterplot  between  crew  performance  sum  on  the  x-axis  and 
T^RM  process  rating  on  the  y-axis.  As  is  evident  from  the  figure,  the  poorest  performing  crews 
did  indeed  have  the  lowest  overall  T^RM  process  rating  whereas  the  highest  rated  crews  received 
the  largest  mission  performance  sums.  The  intermediate  values  also  behave  in  a  similarly  ordered 
fashion. 

Because  ratings  have  a  tendency  to  bunch  in  the  middle  of  the  scale,  we  performed  a 
supplemental  analysis  to  determine  if  an  even  stronger  (i.e.,  more  linear)  process-performance 
relationship  would  materialize  if  rank  orders  were  used.  Recall  that  rank  ordering  formed  part  of 
the  logic  of  the  Povenmire  et  al.  (1989)  methodology,  and  we  wanted  to  determine  if  a  similar 
trend  would  be  found  here.  To  that  end,  our  SME-researcher  reviewed  his  extensive  T-MOT 
ratings  and  observation  notes  for  each  of  the  11  crews.  Based  on  this  review,  he  rank  ordered  the 
11  crews  according  to  an  overall  assessment  of  their  T^RM  process  effectiveness  during  the 
mission.  Independently,  the  psychologist-researcher  reviewed  her  notes  and  T-MPT  rating  data, 
and  formed  a  corresponding  rank  ordering  of  crews  on  the  mission  performance  side.  The 
Spearman  correlation  coefficient  between  the  two  sets  of  rank  orders  was  slightly  larger,  .89,  and 
is  of  course  statistically  significant  (t  =  7.064,  df  =  9,  p  <  .05,  two-tailed).  Figure  12  depicts  a 
scatterplot  of  the  crew  T^RM  and  mission  performance  rank  orders. 
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Figure  12.  Scatterplot  of  Overall  T^RJM  and  Mission  Performance  Rank  Orders 
for  the  1 1  Subject-Crews. 
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Secondary  Results  -  T^RM  Process  and  Mission  Performance  by  Subprocess  and  Mission 
Phase 

Having  established  the  fundamental  relationship  between  T^RM  process  and  mission 
performance,  the  next  step  entailed  pinpointing  the  subprocesses  and/or  phases  in  which  the  rela¬ 
tionship  is  the  strongest.  As  a  first  step,  note  that  the  five  correlations  in  the  top  row  of  Table  10 
depict  the  correlations  between  overall  T^RM  process  and  the  performance  rating  within  each 
mission  phase.  As  denoted  by  the  asterisks,  we  see  that  the  quality  of  the  T^RM  process  is 
significantly  related  to  only  two  of  the  mission  phases,  MP  and  LL.  Though  trending  positive, 
performance  in  the  other  three  phases  was  not  significantly  related  to  T^RM  process.  This  finding 
shows  that  while  the  quality  of  the  T^RM  process  is  important  for  performance,  it  is  not 
uniformly  so,  but  is  instead  localized  in  two  discrete  phases.  It  should  be  noted,  though,  that  the 
large  influence  of  these  phases  may  be  due  to  their  relative  durations  compared  to  the  AD,  AR, 
and  1/E  phases.  That  is,  both  T^RM  subprocess  and  team  mission  performance  ratings  were  based 
on  sampling  behavior  throughout  extended  periods  of  time  (i.e.,  3  hours  for  MP,  blocks  of 
minutes  throughout  mission  execution  for  LL),  thus  providing  more  representative  and  stable 
process  and  performance  measures. 

Examination  of  the  correlations  in  the  next  two  rows  of  Table  10  shows  the  influence  that 
phase-specific  T^RM  process  exerts  on  overall  performance  (second  row)  and  on  phase-specific 
mission  performance  (third  row).  As  seen  in  the  second  row,  four  of  the  phase-specific  T^RM 
process  measures  significantly  predict  overall  mission  performance.  The  only  phase  that  failed  to 
achieve  significance  was  low-level  navigation.  When  considered  together  with  the  discussion 
above,  this  finding  suggests  that  the  influence  of  T^RM  process  on  performance  is  not  localized 
in  a  single  phase,  but  rather  is  present  throughout  the  entire  mission  scenario. 

The  gray-shaded  cells  in  Table  10  depict  the  correlations  that  gauge  the  strength  of  the 
linear  relationship  between  each  of  the  five  overall  T^RM  subprocesses  and  overall  mission 
performance  (i.e.,  the  PerfSum  variable).  As  can  be  seen,  four  of  the  elements  were  statistically 
related  to  mission  performance,  with  only  C3  failing  to  achieve  significance.  The  pattern  of 
results  in  these  cells  clearly  suggests  that  our  decomposition  of  T^RM  into  its  constituent 
subprocesses  was  successfiil  in  identifying  those  factors  having  a  strong  impact  on  performance. 
Interestingly,  the  relationship  between  C3  and  mission  performance  was  near  zero.  In  the 
concluding  section,  we  will  discuss  the  implications  of  this  finding  for  interpreting  some  of  the 
previous  CRM  literature. 

Tertiary  Results  -  T^RM  Subprocess  and  Mission  Performance  by  Mission  Phase 

The  lower  right  quadrant  in  Table  10  (outlined  in  bold)  breaks  down  the  analysis  still 
further,  as  it  comprises  a  5  x  5  matrix  of  correlations  between  each  phase-specific  T^RM 
subprocess  rating  and  its  corresponding  phase-specific  mission  performance  rating. 
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While  all  of  the  correlations  were  positive,  and  some  even  quite  large,  only  the  correlation 
between  the  rating  of  TE  and  mission  performance  during  the  AR  phase  proved  significant  (t  = 
4.97,  df  =  9,  p  <  .001,  one-tailed).  Though  our  statistical  tests  cannot  substantiate  this,  there  is 
some  evidence  in  the  table  to  show  that  the  T^RM  subprocesses  exhibit  differential  effects  across 
mission  phases.  One  line  of  evidence  for  this  differential  sensitivity  comes  fi'om  a  series  of 
multiple  regression  analyses  (MRA)  that  we  performed  across  the  five  phases.  In  each  case,  the 
analysis  showed  that  beyond  the  first  or  second  highest  correlated  T^RM  subprocess  (e.g.,  TE  for 
AR),  not  much  variance  is  accounted  for  by  including  additional  subprocesses  in  MRA  equations 
to  predict  mission  phase  performance. 

This  trend  is  also  evident  in  the  pattern  of  correlations  themselves.  As  we  examine  each 
column,  we  first  see  that  the  T^RM  subprocess  most  strongly  associated  with  the  MP  phase  is 
SA,  with  a  smaller  contribution  by  TM.  The  main  T^RM  subprocess  associated  with  LL  is  FA, 
with  some  impact  of  SA.  The  T^RM  subprocess  most  strongly  associated  with  AD  performance 
is  TM,  although  SA,  TE,  and  FA  also  exhibited  high  positive  correlations.  Besides  TE,  other 
T^RM  subprocesses  that  influenced  AR  performance  were  TM  and  FA.  Finally,  I/E  performance 
was  associated  with  fairly  large  (though  nonsignificant)  correlations  by  all  of  the  subprocesses 
save  C3.  Indeed,  as  can  be  seen  in  the  bottom  row  of  Table  10,  C3  did  not  have  any  correlations 
above  .40  with  any  of  the  mission  phases.  This  is  consistent  with  the  low  correlation  that  the  C3 
subprocess  overall  rating  exhibited  with  overall  mission  performance. 

Taken  together,  these  results  suggest  that  our  subprocess  measures  of  T^RM  have  potential 
to  be  sensitive  barometers  of  team  mission  performance.  The  implications  of  this  sensitivity  for 
training  and  field  assessments  of  team  effectiveness  will  be  discussed  in  the  concluding  section. 

Relationship  of  T^RM  to  MP  and  Team  Mission  Performance 

Although  military  doctrine  is  based  on  the  premise  that  thorough  MP  is  essential  to  mission 
success,  there  is  surprisingly  little  empirical  data  to  support  this  seemingly  self-evident  truth.  We 
therefore  performed  several  special  analyses  to  gauge  the  effect  of  T^RM  on  MP  as  well  as  the 
mediating  effect  that  the  quality  of  MP  may  exert  on  subsequent  mission  execution. 

We  begin  the  discussion  by  noting  that  as  with  overall  mission  performance,  we  did  not 
have  an  overall  rating  of  MP  performance.  Again,  this  reflects  the  diverse  nature  of  the 
behaviors,  skills,  and  information  that  are  employed  throughout  this  phase.  Instead,  our 
psychologist-research  reviewed  her  notes  and  ratings  from  the  T-MPT  to  extract  10  indices  of 
MP  that  were  applicable  to  all  crews.  These  were:  quality  of  crew  briefings,  LN  chart,  RN  chart. 
Pilot  chart,  LN  flight  plan,  RN  flight  plan.  Pilot  flight  plan,  number  of  briefs,  number  of  planning 
products,  and  quality  of  planning  products.  A  Factor  Analysis  was  then  performed  on  these 
indices  to  identify  the  ones  that  maximally  discriminated  among  the  1 1  crews.  Five  measures 
were  extracted  from  this  analysis:  quality  of  crew  briefings,  total  number  of  planning  products, 
quality  of  planning  products,  quality  of  the  LN’s  flight  plan,  and  quality  of  the  RN’s  flight  plan. 
An  overall  MP  performance  score  was  then  established  for  each  crew  by  summing  the  counts  and 
ratings  for  these  five  measures.  This  was  the  index  we  used  for  all  subsequent  analyses  involving 
MP. 
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There  are  several  lines  of  evidence  in  our  data  to  support  the  importance  of  MP  to  CMT. 
First,  recall  that  our  overall  assessment  of  T^RM  process  was  most  strongly  related  to 
performance  during  MP,  as  its  correlation  was  only  slightly  lower  than  the  one  with  overall 
mission  performance  (.78  vs  .86).  Based  on  the  analysis  described  above,  this  means  that  the 
aircrews  who  exhibited  superior  T^RM,  on  average,  gave  better  crew  briefings,  produced  a  larger 
number  of  high  quality  planning  products,  and  had  higher  quality  flight  plans  produced  by  both 
the  LN  and  RN. 

Next,  if  better  T^RM  means  more  effective  MP,  does  that  preparation  translate  into  better 
overall  mission  performance?  In  order  to  make  this  determination,  a  special  analysis  had  to  be 
performed.  Recall  that  our  index  of  overall  mission  performance  was  based  on  the  sum  of  ratings 
from  all  five  phases,  including  MP.  If  we  want  to  determine  the  relationship  between  MP 
performance  and  overall  mission  performance,  we  must  first  remove  MP  rating  from  the  global 
index  sum.  We  accordingly  recomputed  mission  performance  as  the  sum  of  the  ratings  from  the 
four  mission  execution  phases,  i.e.,  LL,  AD,  AR,  and  I/E.  With  this  revised  index,  we  then 
computed  the  correlation  between  MP  performance  and  overall  mission  performance.  The 
correlation  in  this  case  was  .60,  which  approaches  significance  for  a  nominal  alpha  level  of  .01. 

To  get  an  indication  of  the  type  of  T^RM  that  is  associated  with  better  planning,  recall  that 
the  lower  part  of  the  second  data  column  in  Table  10  depicted  the  correlations  between  each 
T^RM  subprocess  and  MP  performance.  Two  subprocesses,  SA  and  TM,  stood  out  as  having  the 
strongest  relationship  to  MP  performance.  The  implications  of  this  relationship  for  training  are 
discussed  in  the  concluding  section. 

Having  established  the  overall  relationship  between  MP  and  mission  performance,  we  next 
wanted  to  determine  the  specific  aspects  of  MP  that  had  the  largest  impact  on  performance.  We 
began  by  comparing  the  42  measures  of  MP  effectiveness  identified  during  a  previous  study 
(Spiker  &  Nullmeyer,  1995b)  with  the  content  items  from  the  MP  segment  of  the  T-MOT.  Of 
these  42,  12  were  judged  as  relevant  to  the  particulars  of  our  study,  i.e.,  a  CMT  scenario 
involving  a  single  WST.  These  are  listed  in  the  left-hand  column  of  Table  1 1 . 

The  1 1  crews  were  then  scored  on  the  extent  to  which  each  of  the  12  measures  were  present 
in  the  observations  of  notable  MP  behaviors  recorded  in  the  T-MOT.  As  a  basic  index  of  MP 
quality,  we  simply  counted  the  number  of  different  measures,  out  of  12,  that  were  represented  by 
one  or  more  behaviors  recorded  in  the  T-MOT.  Examples  of  behaviors  that  would  be  assigned  to 
each  measure  are  shown  in  the  right  hand  column  of  Table  11.  Since  negative  behaviors  were 
also  recorded  in  the  T-MOT,  a  measure  was  scored  as  a  -1  if  it  contained  only  instances  of 
unacceptable  performance.  Theoretically,  scores  could  range  from  -12  to  +12.  The  observed 
range  in  our  sample  of  1 1  crews  was  -1  to  12.  We  then  correlated  these  derived  scores  with  the 
crew’s  rank  order  on  overall  mission  performance.  A  fairly  sizable  and  significant  correlation, 
.71,  was  observed  (t  =  3.02,  df  =  9,  p<  .01).  Thus,  crews  who  exhibited  positive  behaviors  on  a 
larger  number  of  these  measures  of  effectiveness  performed  significantly  better  during  the 
mission. 
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Table  11.  Measures  of  Mission  Preparation  Effectiveness 


Measure  of  Mission 

Preparation  Effectiveness 

Representative  Data  Item 
from  the  T-MOT 

All  planning  personnel  are  effectively  utilized. 

AC  asked  all  crewmembers  for  “what  you  need  to  do  your 
job”  and  then  got  it  for  them. 

A  timeline  is  established  for  managing  the  planning 
process. 

AC  told  crewmembers  when  they  had  to  be  completed  with 
their  planning  tasks  in  time  for  the  crew  briefing. 

Precise  times  are  determined  for  accomplishing  the 
key  mission  events. 

Planned  AR  control  time  and  route  backwards  from  the  AR 
control  point.  Determined  optimal  takeoff  time  from  these. 

High-quality  crew  briefings  are  given  during  various 
stages  of  planning. 

After  each  crewmember  briefs,  the  AC  adds  final  comments 
for  the  crew’s  consideration. 

Planning  crew  achieve  an  in-depth  awareness  of 
threat  capabilities  along  the  route. 

To  avoid  threats,  crew  planned  to  fly  very  low  altitude, 
terrain  mask,  and  high  speed  (as  necessary)  maneuvering. 

Ihe  plan  is  developed  to  an  appropriate  level  of 
detail. 

FE  and  CSU  prepared  the  evasion  plan  of  action  (note:  a 
level  of  detail  not  provided  by  many  of  the  crews). 

All  information  sources  are  checked  for  recency. 

Acj  asked  when  Intel  had  last  been  updated. 

Information  is  cross-checked  for  accuracy  and  the 
plan’s  assumptions  are  aggressively  questioned. 

AC  questions  the  assumptions  made  in  each  crewmember’s 
component  plan. 

Ground  team  and  support  asset  requirements  are 
incorporated  into  the  overall  plan. 

AC  modifies  plan  to  incorporate  considerations  of 
helicopters  for  the  transload. 

Mission  essential  equipment  is  well  thought  out  and 
incorporated  into  the  plan. 

Crew  listed  the  minimum  equipment  needed  to  accomplish 
the  mission,  such  as  INS,  chaff,  flares,  etc. 

Planning  assumptions  are  subject  to  extensive  “what 
iffmg.” 

Crew  planned  to  “bump  up”  their  airspeed  if  they 
encountered  threats  during  the  AR. 

Planners  incorporate  their  real  world  experience  into 
the  planning  process. 

Crewmembers  related  their  own  experiences  in  the  AO  as 
they  developed  the  execution  plan. 

Taken  together,  these  analyses  support  the  view  that  (a)  good  MP  is  indeed  strongly  related 
to  better  mission  execution,  and  (b)  superior  SA  and  TM  are  good  indicators  of  the  types  of 
crews  who  will  have  produced  a  superior  mission  plan. 


Determination  of  Team  Mission  Effectiveness 

One  of  the  objectives  of  our  research  program  is  to  identify  the  structural  and/or  behavioral 
charactenstics  of  effective  aircrews.  However,  in  order  to  make  this  identification,  we  must  first 
have  a  way  to  imequivocally  measure  and  then  classify  those  aircrews  who  are  truly  “effective  ” 
Below,  we  describe  an  analytic  technique  that  was  used  to  construct  an  interval  scale  of  aircrew 
effectiveness,  from  which  we  determined  the  most  and  least  effective  aircrews. 

The  analysis  is  based  on  a  class  of  Thurstonian  scaling  methods  are  described  in  Guilford 
(1954).  The  premise  of  these  analyses  is  that  a  set  of  scores  can  be  transformed  into  interval  z- 
scale  if  one  assumes  that  the  underlying  scale  from  which  original  scores  are  derived  is  normally 
distributed.  In  our  case,  we  assume  that  the  rank  orderings  of  the  1 1  aircrews’  T^RM  and  mission 
performance  produced  by  our  two  researchers  are  normally  distributed.  Using  Thurstone’s 
method  of  rank  order,  we  combined  the  two  rank  orders  into  a  single  scale  by  first  summing  and 
then  averaging  the  rank  order  pairs  (process  and  performance)  for  each  of  the  1 1  crews.  We  then 
rescaled  each  average  so  that  the  lowest  crew  was  located  on  the  zero-point  of  the  scale. 
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The  resulting  scale  of  effectiveness  for  our  11  crews  is  depicted  in  Figure  13.  Scale  values 
range  from  a  low  of  0,  for  Crew  #10,  to  a  high  of  3.52  for  Crew  #13.  The  scale  values  may  be 
interpreted  much  as  one  would  view  any  z-score.  For  example,  looking  at  Figure  13,  one  may  see 
that  Crew  #9  was  2  units  more  effective  than  was  Crew  #10.  Similarly,  Crew  #12  was  another 
full  unit  more  effective  than  Crew  #9.  In  each  case,  then,  we  have  defined  effectiveness  as  a 
combination  of  T^RM  and  mission  performance. 
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Figure  13.  Interval  Scale  of  Effectiveness  and  Type  of  Orientation 
for  the  1 1  Participating  MC-130P  Aircrews. 

Armed  with  this  interval  scale  information,  it  is  a  fairly  simple  matter  to  establish  criteria 
for  labeling  ineffective,  average,  and  effective  aircrews.  These  have  been  provided  in  Figure  13. 
Here,  we  see  that  four  crews  (#3,  5,  10,  11)  have  an  effectiveness  index  less  than  1,  and  have 
been  classified  as  “less  effective”  crews.  On  the  upper  end,  three  crews  (#7,  12,  13)  have  scale 
values  well  above  2,  and  thus  are  labeled  “more  effective.”  The  remaining  four  crews  are  in  the 
middle,  and  are  accordingly  referred  to  as  “average.”  In  the  next  subsection,  we  base  our 
determination  of  the  characteristics  of  more  effective  and  less  effective  crews  on  these  groupings. 

To  shed  further  light  on  the  crews’  distinguishing  characteristics,  each  crew  in  Figure  13 
has  been  labeled  as  “M,”  “T,”  or  “Tr.”  These  labels  refer  to  the  type  of  orientation  that  crews 
exhibited  during  MP  and  were  assigned  by  the  SME-researcher  prior  to  the  mission  execution 
stage.  Specifically,  the  labels  refer  to  a  Mission,  Task,  or  Training  orientation,  respectively. 
Crews  with  a  Mission  orientation  were  always  mindful  of  time  requirements  to  complete  MP, 
and  did  not  get  sidetracked  with  interpersonal  tasks.  On  the  other  hand,  crews  that  maintained  a 
Task  orientation  throughout  were  more  apt  to  get  bogged  down  in  details  of  MP,  even  if  it  meant 
reducing  time  for  crew  briefings  at  the  end  of  the  phase.  Finally,  crews  that  exhibited  a  Training 
orientation  were  always  mindful  that  the  mission  for  which  they  were  preparing  involved  a 
training  scenario,  so  were  less  likely  to  “  go  the  extra  mile”  to  complete  the  tasks  required  for  a 
thorough  job  of  MP.  As  can  be  seen  in  Figure  13,  two  of  the  three  best  crews  exhibited  a  Mission 
focus  where  two  of  the  four  weakest  crews  were  characterized  as  maintaining  a  Training 
orientation.  The  average  crews  represent  a  mix  of  the  three  types  of  crew  orientation. 
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Characteristics  of  Effective  Crews  -  Preliminary  Assessments 


As  a  prelude  to  developing  a  training  intervention  to  promote  more  effective  T^RM,  we 
must  first  determine  those  characteristics  that  are  reliably  associated  with  effective  crews.  In  the 
following  subsection,  we  present  the  findings  from  preliminary  analyses  of  our  data.  We  use  the 
term  “preliminary”  because  more  in-depth  analysis  on  a  larger  data  set  will,  we  think,  be  needed 
to  fully  delineate  the  more  salient  characteristics.  Nevertheless,  the  analyses  conducted  thus  far 
highlight  several  directions  where  future  research  is  needed. 

Structural  Characteristics  of  Effective  Crews  -  Team  Demographic  Variables 

Due  to  the  small  number  of  crews  in  this  research  effort,  it  was  difficult  to  identify  any 
particular  demographic  variable  having  a  significant  relationship  with  effectiveness. 
Nevertheless,  several  interesting  features  are  worth  highlighting  in  the  comparisons  below,  and 
should  receive  further  study  during  follow-up  research.  We  leave  statistical  testing  of  these 
elements  to  future  work  where  a  larger  sample  size  may  be  obtained. 

Squadron  Affiliation.  Visual  inspection  of  the  data  suggests  that  a  crew’s  predominant 
squadron  affiliation  is  perhaps  one  of  the  strongest  demographic  predictors  of  mission  effective¬ 
ness.  For  reasons  confidentiality  reasons,  specific  squadrons  are  not  identified  here.  However,  as 
seen  in  Figure  14,  crews  fi-om  “A”  Special  Operations  Squadron  (SOS)  tended  to  be  associated 
with  low  mission  effectiveness  scores,  while  crews  from  “D”  SOS  tended  to  be  associated  with 
high  mission  effectiveness. 

Crew  Size.  Another  finding  of  note  is  that  crews  trained  under  “artificial”  sizes  of  either  5 
or  7  crewmembers,  rather  than  the  organic  6-member  crew,  did  not  reach  the  upper  levels  of 
mission  effectiveness.  Indeed,  the  crew  size-mission  effectiveness  plot  displayed  in  Figure  15 
shows  that  there  is  a  continuum  of  mission  effectiveness  across  the  6-member  crews,  suggesting 
that  an  orgamc  crew  size,  by  itself,  does  not  guarantee  superior  mission  effectiveness.  However, 
it  was  only  the  6-member  crews  that  achieved  the  highest  ratings  of  mission  effectiveness. 

Crew  Structure.  Crew  “  structure”  is  a  term  we  use  to  refer  to  the  typical  manner  in  which 
crews  were  organized  to  promote  interactions  and  information  sharing.  Structure  was  captured 
through  careful  analysis  of  the  SME-researchers’  notes  and  diagrams  on  the  T-MOT.  Three 
distinct  types  of  crew  structures  were  identified:  (1)  “Hub-and-spoke”  (see  Figure  16)  crews  as 
represented  by  crew  #7,  #12,  #13;  (2)  crews  with  distinct  dyads  and  triads  initially  but  which 
“came  together”  as  a  crew  (not  necessarily  as  hub  and  spoke)  over  time,  #2,  #11,  #4;  and  (3) 
crews  with  distinct  dyads  and  triads  that  seemingly  “never  came  together”  as  a  crew  over  the 
course  of  training,  as  represented  by  crew  #10,  #9,  #5,  #3,  #6.  Due  to  the  highly  specialized 
skills  required  of  each  crew  position,  it  is  plausible  that  even  the  never-came-together  structure 
might  work  as  the  required  mission  events  could  still  be  accomplished.  However,  it  is  interesting 
to  note  that  the  hub-and-spoke  arrangement  was  characteristic  of  the  three  most  effective 
aircrews.  Moreover,  the  never-came-together  structure  aptly  characterizes  three  of  the  four  least 
effective  crews. 
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Figure  14.  Relationship  between  a  Crew’s  Predominant  Squadron  Affiliation 
and  Mission  Effectiveness. 
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Figure  15.  The  Relationship  between  Crew  Size  and  Mission  Effectiveness. 
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Figure  16  is  a  reproduction  of  the  SME-researcher’s  process  depiction  of  one  of  the  most 
effective  cretvs,  #7,  with  the  arrows  indicating  some  of  the  predominant  communication  patterns 
among  its  crewmembers.  The  other  two  hub-and-spoke  crews  (#12, 13)  also  exhibited  this  same 
basic  structure,  although  they  had  different  patterns  of  observed  interactions  (i.e.,  arrows).  The 
SME  described  such  hub-and-spoke  crews  as  having  a  leader  (usually  the  AC)  who  “weaves  all 

crewmembers  into  a  cohesive  unit.” 


Figure  16.  Hub-and-Spoke  Crew  Structure,  Crew  #7. 


Figure  17  depicts  the  evolving  structure  of  a  crew  that  “came  together”  during  MP.  The 
graphic  depicts  a  snapshot  of  the  crew  structure  at  30-minute  intervals  during  planning.  As  can 
be  seen,  the  initial  stages  of  planning  are  characterized  by  crewmembers  working  separately  (or 
in  pairs).  Over  time,  though,  more  cohesive  groupings  were  developed  based  on  planning 
requirements  and  task  assignments.  By  the  final  stages  of  planning,  all  of  the  crewmembers  were 
working  together  with  the  exception  of  the  FE.  The  latter  had  completed  his  (limited)  duties 
required  for  the  mission  and  was  then  tasked  to  do  other  things  prior  to  the  mission  briefing. 

Miscellcineous  Background  Variables.  In  the  absence  of  any  theoretical  expectations,  we 
collected  and  examined  the  effect  of  a  number  of  miscellaneous  crew  background  variables  from 
the  CBS.  These  included:  average  total  hours  of  crew  flying  experience,  average  total  MC-130P 
flying  hours,  average  number-of-hours  experience  in  the  real-world  AO  of  the  simulated  mission, 
number  of  crewmembers  with  experience  in  the  AO  of  the  simulated  mission,  and  number  of 
CRM  training  hours  of  the  crew.  None  of  these  background  variables  showed  any  systematic 
relationship  (linear  or  otherwise)  to  mission  effectiveness.  Perhaps  most  notable  in  this  list  of 
backgromid  variables  was  CRM  training  experience.  We  examined  average  number  of  prior 
CRM  training  hours  the  crew  had  received,  minimum  number  of  CRM  training  hours  within  the 
crew,  and  maximum  number  of  training  hours  within  the  crew.  In  each  case,  CRM  appeared  to 
exhibit  no  relationship  to  crew  mission  effectiveness.  The  absence  of  any  relationship  to  CRM 
training  is  obviously  disappointing  and  is  a  finding  that  will  be  scrutinized  in  future  work. 
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Behavioral  Characteristics  of  Effective  Crews 


Besides  the  large  number  of  quantitative  ratings  that  were  generated,  we  also  recorded 
extensive  and  detailed  observations  of  T^RM  process  and  performance-related  behaviors  for  each 
of  the  11  crews.  Below,  we  describe  some  behaviors  that  seem  to  characterize  the  three  most 
effective  crews  depicted  in  Figure  13  (#7,  12,  and  13).  We  hope  to  expand  upon  this  preliminary 
analysis  in  a  future  report  on  T^RM. 
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Figiore  17.  Evolving  Structure  of  a  “Crew  that  Came  Together,”  Crew  #2. 
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Recall  that  high  SA  ratings  were  often  associated  with  good  mission  performance.  Some  of 
the  SA  behaviors  from  the  T-MOT  that  accompanied  the  high  ratings  were: 

1 .  More  consideration  of  the  “big  picture;” 

2.  viewing  the  crew  as  only  a  part  of  the  (larger)  team  and  mission; 

3.  extensive  “  what-iffiing,”  especially  of  main  mission  events,  including  input  from  the 
entire  crew; 

4.  willingness  of  the  crew  to  change  their  plan  even  if  it  caused  some  confusion  to  the  crew, 
based  on  the  evolving  mission  and  changing  situation; 

5.  including  explicit  contingency  plans  within  the  pre-mission  briefings,  e.g.,  no  waiting  on 
the  AR  track  if  the  helicopter  is  late  or  cut  the  jumper  if  he  gets  himg;  and 

6.  responding  well  to  their  own  errors  or  changing  conditions. 

FA  was  also  highly  correlated  with  mission  performance,  with  two  key  FA  behaviors 
associated  with  exceptional  crew  mission  performance.  First,  at  least  one  crewmember  overtlv 
and  explicitly  designated  the  duties  to  the  rest  of  the  crew-this  was  done  more  than  once  during 
the  course  of  the  mission.  This  explicitness  was  in  contrast  to  the  more  implicit  designation  of 
duties  that  occurred  in  the  lower  rated  crews. 

Second,  the  exceptional  crews  seemed  to  designate  duties  based  on  crewmember  strengths 
rather  than  position.  For  example,  one  AC  said,  “CP,  you’ll  be  in  charge  of  SOFPARS  (SOF 
Planning  and  Rehearsal  System)  since  you  know  it.”  The  AC  recognized  the  efficiency  of  this 
designation  rather  than  simply  assuming  since  it  was  a  navigator’s  duty,  it  should  therefore  be 
best  completed  by  a  navigator.  Another  example  was  an  AC  who  designated  ^  communication 
responsibilities  to  the  CSO.  Although  communication  is  the  CSO’s  primary  function,  many 
crews  opted  for  the  navigator  and/or  CP  to  support  these  duties  as  well.  This  particular  AC 
seemed  to  realize  the  complexity  of  the  mission  and  the  necessity  to  allocate  this  function  solely 
to  the  CSO,  so  that  the  navigators  and  CP  could  focus  their  attentions  on  other  mission  taskings. 

In  addition,  two  central  TE  behaviors  were  indicative  of  exceptional  crews.  One,  the 
“effective”  crew  responses  to  threats  included  more  than  just  avoidance,  evasion,  and  survival. 
For  example.  Crew  #13  included  a  separate  threat  briefing  among  their  pre-mission  briefs.  This 
briefing  contained  a  detailed  overview  of  the  threat  situation  (especially  “the  shooters”),  and 
included  preplanned  crew  responses  to  the  threats  depending  on  threat  mode  (i.e.,  search, 
acquisition,  locked).  While  some  weaker  crews  did  not  even  include  a  threat  briefing  in  their 
premission  briefings,  other  weak  crews  did.  However,  the  latter  crews  did  not  include  variations 
in  threat  response  based  on  specific  threat  modes  as  did  Crew  #13.  Two,  the  highest  rated  crews 
had  more  extensive  tactical  knowledge  than  the  lesser  rated  crews.  Examples  include  knowing 
the  typical  threats  in  the  AO  as  well  as  understanding  field  (vs.  textbook)  techniques  for  threat 
evasion. 
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Another  superior  example  of  TE  comes  from  Crew  #12.  Their  responses  to  threats 
included:  (a)  asking  if  anyone  on  the  crew  had  a  visual;  (b)  asking  about  aircraft  damage;  (c) 
marking  the  latitude  and  longitude  (lat/long)  of  the  threat  encounter;  (d)  transferring  the  lat/long 
information  to  ABCCC;  and  (e)  suggesting  additional  countermeasures  even  after  successfully 
avoiding  the  threat  (e.g.,  dispatching  an  F- 16  to  destroy  target  “  ’X”  to  ensure  the  safety  of  future 
missions  or  in  the  event  they  were  forced  back  into  its  vicinity  later  in  the  mission). 

TM  is  a  fundamental  feature  of  SOF  MC-130P  missions  and  was  highly  correlated  with 
mission  performance.  The  primary  behavior  that  characterized  effective  crews  in  this  regard  was 
their  overt  time  awareness  and  monitoring  throughout  MP  and  execution.  Exceptional  crews 
would  set  times  for  briefings  and  TOTs  as  well  as  ask  about  their  progress  towards  these  times. 
Within  weaker  crews,  monitoring  of  time  status  was  less  apparent  and  overt,  especially  during 
planning.  As  a  result,  many  weaker  crews  began  briefings  late  or  rushed  through  their  briefings. 

Although  C3  was  not  significantly  correlated  with  overall  mission  performance,  there  were 
still  C3  behaviors  that  clearly  characterized  the  exceptional  crews. 

One,  exceptional  crews  seemed  to  have  a  much  greater  awareness  of  higher  echelon 
mission  players,  a  characteristic  that  is  certainly  related  to  SA.  These  crews  planned  and 
functioned  more  as  a  larger  team,  including  making  requests  for  additional  resources  to  enable 
successful  completion  of  the  mission.  In  effect,  they  were  much  more  “out  of  box”  than  were  the 
weaker  crews. 

Two,  exceptional  crews  seemed  very  conscious  of  how  their  actions  affected  other  team 
players.  For  example.  Crew  #12  talked  extensively  about  countermeasures  they  would  use  on  the 
AR  track  and  consulted  often  with  the  (role-played)  helicopter-representative  as  to  what  he 
would  use.  This  inquiry  is  critical  because  the  AC  and  CP  of  Crew  #12  did  not  want  to  blind  the 
helicopter  whose  crew  was  on  NVGs  by  dispensing  flares.  This  crew  also  did  not  decide  to  use 
the  ramp  or  the  door  for  the  AD  based  only  on  their  time  constraints  and  mission  requirements. 
They  consulted  with  the  (role-played)  PJs  to  determine  what  their  needs  were.  Every  other  crew 
simply  made  the  offloading  decision  based  on  their  own  constraints,  without  any  consideration  of 
the  PJ’s  needs.  Another  effective  crew,  #13,  was  also  acutely  conscious  that  their  actions  needed 
to  be  coordinated  with  other  team  players.  For  example,  the  offload  of  the  General  was  planned 
with  extensive  input  from  the  (role-played)  General.  The  crew’s  AC  asked  the  General  where  on 
the  runway  he  would  like  to  be  offloaded,  the  size  of  the  reception  party,  the  validation  signals  of 
the  reception  party,  what  to  do  if  there  were  hostile  forces  in  the  area,  and  so  on. 

A  few  other  process  behaviors  distinguished  exceptional  crews  from  weaker  crews.  How¬ 
ever,  these  did  not  readily  fall  into  any  of  our  five  T^RM  subprocesses.  The  most  salient  “other” 
behaviors  are  listed  below. 

1.  Exceptional  crews  were  extremely  focused  on  the  mission  with  little  (if  any)  socialization 
during  planning;  there  was  no  chatter  at  all  on  the  intercom  during  mission  execution. 
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2.  Exceptional  crews  tended  to  use  extremely  aggressive  plans. 

3.  Exceptional  crews  were  characterized  as  highly  integrated  crews  versus  fragmented  dyads 
and  triads  (as  previously  described). 

4.  Exceptional  crews  had  ACs  who  worked  to  weave  [the]  crews  together  in  all  mission 
aspects. 

Our  T-MPT-based  performance  observations  were  not  as  extensive  as  our  T^RM  observa¬ 
tions.  Nevertheless,  there  were  still  several  performance  behaviors  that  differentiated  exceptional 
crews  from  weaker  ones.  These  behaviors  included: 

1.  High  quality  and  level  of  detail  of  pre-mission  briefs, 

2.  high  number  and  quality  of  mission  products, 

3.  superb  mission  plans, 

4.  outstanding  time  control,  and 

5.  outstanding  accuracy  for  accomplishing  main  mission  events  (e.g.  drop  on  target). 

In  a  future  report,  we  will  detail  more  precisely  both  the  process  and  performance 
behaviors  associated  with  exceptional  crews.  We  will  also  begin  to  explore  further  the  behaviors 
associated  vvdth  specific  crew  positions  that  hinder  and  help  performance  (Silverman,  Spiker, 
Tourville,  &  Nullmeyer,  in  press).  We  strongly  believe  that  a  systematic  delineation  of  these 
discriminating  behaviors  will  have  a  potentially  important  impact  on  T^RM,  and  hopefully 
should  be  ones  that  could  be  incorporated  into  an  ongoing  training  progreim. 


Supplemental  Analyses 

We  also  explored  the  T^RM  process-performance  relationship  using  the  overall 
performance  assessments  from  the  instructors  (i.e.,  from  the  IRIs)  and  crewmember  self- 
assessments  of  crew  coordination  and  performance  (from  the  TMAQ2s).  Instructor  ratings  of 
crew  performance  were  only  minimally  correlated  with  process  ratings  (r  =  .30),  and 
interestingly,  were  not  at  all  correlated  with  the  mission  performance  sums  described  above  (r  = 
-.05).  Although  these  low  correlations  are  immediately  disconcerting,  a  closer  look  at  the 
instructor  ratings  revealed  that  the  instructors  were  extremely  unlikely  to  use  the  entire  scale 
provided.  Averaged  across  instructors,  the  ratings  ranged  only  from  3.67  to  4.75.  Indeed,  often 
times,  the  comments  instructors  provided  on  the  IRIs  defied  their  own  ratings,  with  negative 
observations  and  assessments  associated  with  average  or  superior  ratings.  Because  of  the 
restricted  scale  range  and  the  observed  disparity  between  instructor  ratings  and  comments,  we  do 
not  view  the  low  correlation  between  the  IRJ  performance  ratings  and  T-MOT  process  ratings  as 
particularly  damaging  to  our  primary  hypothesis. 
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Recall  that  the  TMAQ2s  contained  crewmember  self-ratings  of  crew  coordination  and 
mission  performance.  As  with  our  primary  instnunents,  a  strong  positive  correlation  was  found, 
r  =  .86  (t  =  6.025,  df  =  9,  p  <  .05,  two-tailed).  Admittedly,  these  two  sources  of  data  are  not 
independent,  as  both  ratings  came  from  the  same  crewmembers.  Thus,  it  means  that 
crewmembers  who  felt  their  crew  did  well  on  the  mission  were  also  sure  that  their  crew 
coordination  was  good.  Nevertheless,  this  finding  provides  secondary  validation  concerning  the 
strength  of  the  relationship  between  T^RM  and  combat  mission  performance  as  perceived  by  the 
crews  themselves. 


General  Training  Observations 

Before  shifting  to  the  conclusions  section,  we  offer  a  number  of  general  observations 
regarding  MC-130P  ART  in  this  final  subsection.  These  observations  are  not  based  on  the  T- 
MOT  or  T-MPT,  but  come  from  our  experiences  attending  multiple  days  of  ART,  reviewing 
ART  course  materials  (e.g.,  AFSOCR  51-130),  and  holding  discussions  with  ART  instructors 
and  students.  The  observations  primarily  pertain  to  the  conduct  of  ART  and  are  categorized  into 
five  primary  topics:  ART  curriculum  strategy,  training  objectives,  CRM  training  and  delivery, 
mission  planning,  and  debriefing  procedures. 

ART  Curriculum  Strategy 

During  the  course  of  our  data  collection,  we  observed  a  major  change  in  the  MC-130P  ART 
because  a  multiship  simulation  capability  became  operational  at  the  58  TRSS.  On  Day  4  of 
training,  MC-130P  crews  would  typically  perform  systems  and  procedures  (see  Table  2)  in  the 
WST,  which  included  extensive  malfunction  practice  for  the  pilots  and  FEs.  However,  about 
halfway  into  our  study,  this  shifted  to  accommodate  the  multiship  simulator  mission  which 
incorporated  MH-53J  and  MH-60G  helicopter  teams.  The  Day  4  multiship  mission  was  similar 
in  many  ways  to  the  Day  5  mission,  including  a  limited  MP  session,  an  AR,  tactical  daytime  LL, 
and  an  I/E.  A  large  proportion  of  the  MC-130P  crews  and  instructors,  although  enamored  with 
the  networking  capability,  felt  that  having  both  the  Day  4  and  Day  5  mission  precluded  the 
opportunity  to  provide  focused  training  on  aircraft  systems  and  EPs. 

Training  Objectives 

MC-130P  ART  course  materials  presently  lack  reference  to  any  CRM  training  objectives 
and  guidance  regarding  the  manner  or  matter  of  skills  training  to  be  employed.  AFSOCR  51-130 
provides  only  an  events-based  requirement  for  ART;  no  specification  for  a  skills-based  training 
curriculum  or  course  content  is  suggested.  A  blueprint  is  provided  for  who  (which  crewmembers) 
should  attend  ART,  what  training  events  and  how  many  of  these  events  should  be  signed  off  on 
an  annual  basis  by  completing  ART,  and  the  (minimum)  number  of  hours  to  be  logged  in  the 
simulator.  There  is  a  definite  lack  of  specific,  criterion-referenced  training  requirements,  and  we 
observed  a  great  deal  of  latitude  in  the  execution  of  ART  for  some  of  the  academic  and  simulator 
periods. 
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To  illustrate,  currently  APSOCR  51-130  requires  students  to  perform  (on  an  aimual  basis) 
an  AR  rendezvous  and  join-up  in  the  simulator.  There  is  also  a  specification  to  perform  tactical 
low-level  operations  in  the  simulator.  The  main  intent  of  these  requirements,  however,  is  to 
simply  provide  opportunities  to  perform  certain  training  requirements  so  that  these  may  be 
“logged  off.”  The  squadron  does  not  need  to  generate  local  flying  sorties  to  meet  these  same 
requirements.  Again,  AFSOCR  51-130  provides  no  guidance  on  how,  or  at  what  level  of 
competency,  these  requirements  should  be  met.  Instead,  the  manner  of  training  is  left  to  the 
simulator  instructors  who  must  work  to  combine  the  greatest  number  of  events  into  a  limited 
number  of  simulator  training  hours  in  order  to  provide  the  maximum  opportunity  to  sign  off  a 
large  number  of  training  events. 

Despite  these  problems,  the  overall  level  of  instruction  was  outstanding.  However,  perhaps 
due  to  lack  of  specific  training  objectives,  several  observations  regarding  MC-130P  simulator 
instructors  are  noteworthy. 

A  key  observation  regarding  instructors  was  variability,  not  just  in  terms  of  style  which  will 
always  be  there,  but  also  in  terms  of  the  training  they  provided.  This  was  particularly  evident 
with  threat  management  and  inputting  unplaimed  threats  into  the  scenario.  We  observed  several 
instructors  insert  unplanned  threats  into  the  scenario,  while  others  did  not  modify  the  threat 
laydown  at  all.  Inserting  unplanned  threats  has  a  dramatic  effect  on  the  scenario’s  level  of 
difficulty  and  most  crews  responded  favorably  to  added  threat  exposure.  One  instructor  technique 
that  seemed  particularly  beneficial  was  the  use  of  additional  implanned  threats  toward  the  end  of 
the  scenario  after  the  primary  mission  execution  phases  (LL,  AD,  AR,  and  I/E)  had  been 
completed.  The  scenario  in  its  entirety  is  too  long  for  the  amount  of  simulator  time  allotted  and 
instructors  generally  provide  system  malfunctions  toward  the  end  of  the  time  available  to  force 
the  aircrew  to  make  real-world,  mission-ending  decisions.  Other  instructors  simply  stop  the 
mission  wherever  it  happens  to  be  after  four  hours  have  elapsed.  However,  one  instructor  kept 
inserting  more  and  more  threats  for  the  crew  to  respond  to  which  forced  a  mission-ending 
decision,  but  also  provided  the  crew  with  invaluable  threat  exposure.  This  crew  was  extremely 
enthusiastic  about  the  additional  threat  training  they  received. 

There  were  also  variations  in  terms  of  the  amount  of  instructor  responsiveness  to  the  needs 
of  the  crew  and  the  dynamics  of  particular  crews.  One  outstanding  instructor  with  extensive  past 
CRM  training  experience  observed  the  entire  MP  of  an  aircrew.  He  also  had  previous  experience 
with  the  two  pilots  who  were  present.  One  of  the  pilots  was  a  long-time  CP  who  apparently  was 
reluctant  to  upgrade  to  AC,  despite  his  capability  and  skill  level.  After  the  MP  session  was  over, 
the  IP  retained  the  other  instructors  momentarily  to  inform  them  of  a  change  in  the  scenario  he 
was  going  to  introduce  for  this  crew.  Towards  the  end  of  the  mission,  he  was  planning  to 
simulate  the  incapacitation  of  the  AC,  and  the  CP  was  going  to  have  to  take  control  of  the  aircraft 
and  crew.  He  explained  that  this  would  give  the  CP  the  practice  he  needed  to  boost  his 
confidence,  and  it  would  be  a  unique  CRM  exercise  for  the  rest  of  the  crew.  This  kind  of  respon¬ 
siveness  was  also  observed  intermittently  at  a  crewmember  level.  For  example,  instructor  CSOs 
provided  more  distracter  messages  to  more  highly  skilled  CSOs.  At  both  the  crewmember  and 
crew  level,  this  sort  of  training  specificity  seemed  highly  effective. 
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Finally,  interviews  with  instructors  provided  important  insights  into  local,  environmental 
constraints  that  dictate  much  of  the  training  they  can  provide.  Instructors  have  noted  that  they  are 
not  empowered  to  give  the  type  of  feedback  and  critique  of  ART  students  they  feel  is  needed. 
Also,  as  described  above,  the  local  training  emphasis  (especially  for  ART)  is  simply  to 
accomplish  a  certain  number  of  events  (vs  training  and  evaluation)  before  aircrews  return  to  their 
squadrons.  The  number  of  events  logged  off  during  ART  is  a  way  for  operational  squadrons  to 
determine  the  “value”  of  ART.  None  of  the  instructors  endorsed  this  view,  but  instead,  felt 
unduly  constrained  by  the  demand  to  accomplish  a  large  number  of  training  events. 

CRM  Training  and  Delivery 

In  terms  of  the  CRM  training  provided  during  ART,  three  general  observations  stand  out. 
One,  the  materials  covered  were  somewhat  repetitive  with  those  covered  during  Mission 
Qualification  (MQ)  training.  Instructors  present  essentially  an  abridged  version  of  the  two-day 
MQ  CRM  course.  To  add  diversity  to  the  material,  instructors  try  to  facilitate  a  more  open 
discussion  forum  related  to  the  issues  presented.  This  involves  students  relaying  their  field 
experiences  of  successes  or  failures  associated  with  the  different  CRM  principles.  In  theory,  this 
seems  to  be  a  good  technique.  However,  our  observations  revealed  that  crewmembers  are  very 
reluctant  to  share  stories  in  the  current  ART  CRM  training  environment,  resulting  in  a  very 
“dry” presentation  of  materials  already  viewed  (at  least)  once  by  ART  students.  Instructors 
suggested  that  CRM  lecture/discussion  sections  of  ART  have  become  too  large  to  promote  open 
discussions.  That  is,  the  CRM  lecture/discussion  portion  of  ART  can  include  as  many  as  six 
different  crews  from  four  different  airfitimes,  meaning  up  to  40  students  being  trained  at  once.  It 
is  an  atmosphere  that  does  not  seem  to  promote  self  or  crew  disclosure  about  mistakes  they  have 
made.  Instructors  have  commented  about  their  fixistration  with  the  situation  and  their  feeling  of 
the  limited  utility  of  simply  re-presenting  material  fi-om  MQ  without  an  open  forum.  With  classes 
so  large,  instructors  do  not  feel  that  they  have  much  of  a  choice.  Anecdotally,  instructors  have 
commented  on  the  composition  of  the  class  as  well,  indicating  that  the  CRM  issues  a  helicopter 
crew  deals  with  may  be  different  than  those  of  a  fixed-wing  operator,  especially  at  higher 
qualification  levels. 

Two,  there  is  a  striking  discontinuity  between  the  content  of  the  two-hour,  crew 
coordination  lecture  and  the  training  received  during  the  rest  of  ART.  In  the  lecture,  CRM  was 
described  as  a  set  of  abstract  processes  (e.g.,  advocacy,  SA,  communication)  whose  relevance  to 
training  events  in  the  simulator  was  not  made  explicit. 

Finally,  there  was  not  much  feedback  provided  by  instructors  to  crews  on  their  crew  coordi¬ 
nation.  Not  only  were  the  CRM  principles  not  reintroduced  during  the  subsequent  days  of  ART, 
very  little  reinforcement  (or  punishment)  was  provided  to  aircrews  on  CRM  behaviors  exhibited 
during  the  Day  5  “MOST  (Mission-Oriented  Simulator  Training)  mission.”  The  feedback  that 
was  observed  rarely  went  beyond  statements  like,  “your  crew  coordination  was  good.”  Casual 
discussions  with  instructors  suggest  that  this  is  also  a  problem  for  MQ  CRM  training.  They  cite 
the  AFSOCR  51-130  which  indicates  that  the  goal  of  CRM  training  for  MQ  is  “familiarization.” 
However,  it  is  unclear  what  this  entails.  Is  it  familiarization  with  the  terms  and  principles  or  the 
practices  of  CRM? 
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Mission  Planning 


Several  observations  were  made  regarding  MP  training  and  content  during  ART.  As  we 
have  stressed,  MP  skills  are  critical  components  of  individual  crewmember  combat  capability 
and  eventual  mission  success.  ART  is  an  ideal  arena  to  practice  and  hone  planning  skills,  as  well 
as  to  try  out  new  mission  planning  tools  (e.g.,  SOFPARS),  especially  on  Day  5  of  ART  where  a 
full-mission  team  complement  and  a  real-world  mission  are  used. 

MC-130P  ART  provides  crews  with  four  hours  of  planning  time  for  Day  5’s  mission, 
access  to  state-of-the-art  tools  like  SOFPARS,  several  sources  of  information  on  the  latest  threat 
capabilities,  access  to  copiers  and  real-world  planning  products  and  tools,  and  instructors  for 
each  crew  position.  Nevertheless,  we  observed  several  deviations  from  this  ideal.  If  there  were 
unforeseen  time  constraints  or  scheduling  conflicts,  MP  usually  suffered  the  most.  For  example, 
although  instructors  were  always  present  for  the  entire  mission  execution  phase,  they  were  often 
given  concurrent  duties  that  conflicted  viith  the  MP  phase  of  training.  This  hindered  their  ability 
to  provide  feedback  to  the  crew  or  individual  crewmembers  for  this  critical  mission  phase.  Also, 
ART  scheduling  difficulties  were  noted  three  times  during  the  study,  and  each  time  the  MP  stage 
was  the  only  stage  of  the  mission  that  was  affected.  On  two  of  these  occasions,  the  MP  time  was 
reduced  by  an  hour  and  once  it  was  cut  in  half  Aside  from  markedly  reducing  the  effectiveness 
of  MP  training,  these  cuts  send  a  potentially  devastating  message  to  aircrews  about  the  lack  of 
importance  of  MP  to  overall  mission  success.  Additionally,  and  perhaps  related  to  the  newness  of 
SOFPARS,  this  tool  was  only  available  to  about  half  the  crews  that  we  observed.  The  variable 
that  seemed  to  dictate  availability  of  SOFPARS  to  aircrews  was  instructor  familiarity  with  its 
use.  Finally,  and  perhaps  inevitably,  there  were  several  complaints  by  aircrews  that  the  tools  they 
normally  (i.e.,  in  the  field)  use  for  MP  (e.g..  Integrated  Many  on  Many  System  or  IMOMS)  were 
not  available,  which  they  felt  made  the  MP  process  somewhat  artificial. 

Despite  SOF  crewmember  schedules  and  the  temptation  to  coast  through  an  easy  week  of 
training,  we  observed  crews  who  were  highly  motivated  and  wanted  to  achieve  the  maximum 
gaining  value  for  time  they  spent  at  KAFB.  However,  there  were  some  exceptions,  mostly  by 
individual  crewmembers.  However,  there  was  one  crew  that  seemed  to  feel  that  MP  was  not  an 
essential  component  of  their  week  of  ART.  This  crew  basically  “blew  off’  MP  and  performed 
worse  than  other  crews  during  the  mission  execution  stage.  In  contrast,  other  crews  worked 
vigorously  on  the  MP  tasks,  with  two  crews  even  working  through  lunch  to  prepare  for  the 
mission. 

Debriefing  Procedures 

There  is  nearly  universal  agreement  among  MC-130P  aircrews  about  the  value  of  debriefs 
for  training  and  crew  coordination  (Spiker  et  al.,  in  press).  However,  mission  debriefs  rarely  went 
beyond  critiques  of  flight  procedures  and  filling  out  end-of-course  paperwork.  Debriefing  proce¬ 
dures  and  topics  were  strikingly  inadequate.  For  example,  although  there  is  a  detailed  script  of 
events  and  procedures  for  the  mission  execution  stage,  the  topics  and  requirements  of  the  debrief 
are  loosely  organized.  Many  instructors  wrote  down  particular  events  and  exchanges  from  the 
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mission  to  discuss  during  the  debrief,  but  even  those  were  often  given  short  consideration  as  time 
had  a  tendency  to  run  out.  Also,  videotaping  have  been  cited  as  an  invaluable  CRM  training 
medium  (e.g.,  Diehl,  1995).  Although  the  capability  exists  within  the  MC-130P  WST,  we  never 
observed  its  use.  The  MC-130P  WST  also  has  various  crew  performance  pages  that  can  be 
printed  out  and  used  to  aid  instructor  debriefs.  Over  the  entire  course  of  our  study  (including 
baseline  observations),  we  only  observed  one  instructor  use  this  medium.  Notably,  the  crew/pilot 
who  received  a  debrief  using  this  tool  was  visibly  excited  about  the  documentation,  and  after 
declassifying  the  printout,  took  it  with  him. 

Finally,  MC-130P  ART  suffers  like  many  areas  of  the  USAF  in  terms  of  the  pressures  on 
the  personnel.  The  instructors  have  numerous  responsibilities,  only  one  of  which  is  ART.  As  we 
have  already  mentioned,  ART  instructors  were  often  “double  booked”  and  unable  to  attend 
certain  portions  of  training.  This  potentially  reduces  their  ability  to  guide  students  and  provide 
observations  and  feedback. 


CONCLUSIONS 

We  conclude  this  report  with  a  series  of  discussions  concerning  future  directions  for  CRM- 
related  training,  research,  and  implementation.  We  begin  by  summarizing  the  major  results  of  the 
study,  with  primary  emphasis  on  areas  where  prior  research  either  was  or  was  not  replicated. 
Next,  we  discuss  some  of  the  directions  that  future  CRM  research  should  take  given  the  present 
results.  Then  we  describe  the  implications  of  this  research  for  future  USAF  R&D,  particularly  as 
it  pertains  to  joint  service  training  and  distributed  mission  training  (DMT).  Finally,  we  discuss 
the  implications  of  our  findings  for  improving  the  delivery  of  CMT,  both  at  the  58  TRSS  and 
beyond. 


Summary  of  Major  Findings 

The  most  salient  finding  of  this  research  was  the  replication  of  the  Povenmire  et  al.  (1989) 
observation  that  crew  coordination  process  is  a  significant  predictor  of  tactical  mission 
performance.  In  our  study,  this  relationship  was  supported  by  a  correlation  of  .86  (or  .89  if 
independent  process  and  performance  rank  orders  were  compared).  There  has  been  limited 
support  for  this  finding  in  previous  empirical  investigations  (e.g.,  Thornton  et  al.,  1992).  There 
have  also  been  numerous  descriptions  of  the  relationship  between  crew  coordination  and 
performance  in  the  context  of  accident  investigations  and  safety  violations  (e.g.,  Diehl,  1989; 
Predmore,  1991;  or  Taggart,  1993).  However,  direct  empirical  support  for  this  relationship  has 
been  scarce,  especially  in  tactical  environments. 

The  sigmficant  correlation  between  T^RM  process  and  mission  performance  then  afforded 
us  the  opportumty  to  explore  the  relationships  of  particular  subprocesses,  mission  phases,  and 
ultimately,  specific  behaviors  to  overall  performance  and  T^RM.  These  detailed  analyses  provide 
us  with  the  critical  information  required  to:  (a)  offer  feedback  for  the  CRM-CMT  training 
curricula,  (b)  improve  our  data  collection  instruments  (e.g.,  by  making  them  shorter,  more 
efficient,  and  portable),  and  (c)  further  our  research  efforts. 
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Some  of  the  highlights  of  the  detailed  analyses  of  the  process-performance  relationship  are 
summarized  below.  For  each  main  finding,  we  try  to  establish  how  it  fits  in  to  past  CRM 
research.  In  addition,  we  offer  potential  explanations  for  some  of  the  findings  based  on  our 
knowledge  of  the  MC-130P  SOF  mission,  interviews  with  SMEs,  and  skill  levels  of  ART 
attendees. 

PRM process  in  the  MP,  AD,  AR,  and  I/E  mission  phases  were  all  highly  correlated  with  total 
mission  performance. 

This  result  provides  empirical  confirmation  that  crew  coordination-T^RM  process-is 
critical  to  performance  throughout  the  entire  mission.  Past  CRM  training  evaluation  research  has 
focused  on  student  attitudes,  learmng  motivations,  and  crew  behaviors  during,  immediately 
following,  or  any  time  after  the  training  occurred  (Gregorich  &  Wilhelm,  1993).  Many  of  those 
who  have  focused  on  CRM  evaluation  during  training  or  MOST  missions  (like  us)  have  assessed 
CRM  behaviors  at  several  points  throughout  the  entire  mission  (e.g.,  Fowlkes,  Lane,  Salas, 
Franz,  &  Oser,  1994;  Helmreich,  Wilhelm,  Kello,  Taggart,  &  Butler,  1990).  Fowlkes  et  al. 
(1994),  used  their  TARGETS  (targeted  acceptable  responses  to  generated  events  or  tasks) 
methodology  to  assess  CRM  across  three  segments  of  a  military  helicopter  cargo  mission.  They 
found  distinctions  in  “CRM  performance”  across  the  segments,  noting  CRM-performance 
advantages  for  those  crews  that  had  ACT  versus  those  that  did  not.  Similarly,  the  Line/LOS 
(Line  Operation  Simulation)  Checklist  (Helmreich  et  al.,  1990)  is  designed  to  assess  CRM  across 
four  mission  phases  (predeparture,  take-off  and  climb,  cruise,  and  descent/approach  landing). 
Clearly,  past  researchers  recognized  the  importance  of  CRM  assessment  for  the  entire  range  of 
mission  phases.  But  until  now,  there  has  been  a  surprising  lack  of  data  that  ties  phase-specific 
CRM  evciluation  to  mission  performance. 

MP  and  LL  performance  showed  the  highest  association  with  overall  t^RM  quality. 

We  found  performance  in  the  MP  and  LL  phases  to  have  the  strongest  association  with 
crew  T^RM.  This  is,  we  believe,  due  to  the  relatively  long  durations  of  these  mission  phases 
compared  to  the  other  phases  (AD,  AR,  and  I/E).  MP  behaviors  are  sampled  over  the  course  of 
the  entire  3 -hour  planmng  session,  in  the  form  of  evaluations  of  prepared  charts,  flight  plans, 
number  of  briefs,  quality  of  briefs,  etc.  Similarly,  LL  performance  is  assessed  at  various  intervals 
throughout  the  entire  mission.  As  such,  MP  and  LL  perhaps  offer  the  most  stable  measures  of 
crew  performance. 

Both  MP  process  and  MP  performance  had  a  large  impact  on  total  mission  performance. 

The  importance  of  MP  for  the  successful  execution  of  military  aircraft  missions  is  a  corner¬ 
stone  of  AF  doctrine  and  training  (Hunt,  1993).  While  it  stands  to  reason  that  more  effective  MP 
should  result  in  better  mission  performance,  until  now,  there  has  been  little  empirical  evidence  to 
support  this  assumption.  Studies  of  team  mission  performance  have  typically  not  included  the 
MP  period  as  part  of  the  measurement  process.  This  is  perhaps  because  MP  involves  a  host  of 
cognitive  processes  which  are  inherently  more  difficult  to  measure  than  mission  performance 
(Taylor,  1993).  Some  investigators,  on  the  other  hand,  have  inferred  the  type  of  MP  processes 


59 


that  must  have  occurred  and  anecdotally  linked  those  to  mission  performance  (e.g.,  Thornton  et 
al.,  1992).  Our  study  provides  empirical  evidence  that  more  effective  MP  is  associated  with 
better  mission  performance.  The  relevant  correlations  ranged  from  .60  to  .78.  The  SME’s  rating 
of  crew  coordination  process  during  MP  was  related  to  the  second  researcher’s  rating  of  overall 
crew  mission  performance  during  MP.  Additionally,  ratings  of  the  products  generated  during  MP 
were  a  significant  predictor  of  crew  performance  during  the  four  mission  execution  phases. 

SA,  TE,  FA,  and  TM  were  all  significantly  correlated  with  total  mission  performance,  while 
C3  was  not 

Perhaps  due  to  our  rather  extensive  pilot  work,  including  interviews  with  SMEs  and  a 
review  of  recent  CRM  literature  (see  Spiker  et  al.,  1996  for  details),  all  but  one  of  our  selected 
T^RM  subprocesses  correlated  with  mission  performance.  Where  possible,  we  selected 
subprocesses  that  made  contact  with  past  CRM  and  pilot/crew  performance  research  (e.g.,  SA 
and  C3),  while  providing  as  much  of  a  tactical  focus  as  possible  (e.g.,  TE). 

SA.  In  the  last  decade,  there  has  been  a  veritable  explosion  of  research  on  SA  measurement 
(e.g..  Bell  &  Waag,  1995;  Endsley,  1995a),  definition  (e.g.,  Endsley,  1995b;  Flach,  1995),  and 
use  (e.g.,  Carretta,  Perry,  &  Ree,  1996;  Endsley,  1995b).  However,  there  have  been  some  ambi¬ 
guities  in  distingmshing  S  A  from  performance,  and  accordingly,  whether  it  is  simply  a  phenome¬ 
non  to  be  described  or  it  is  a  truly  causal  entity  (Flach,  1995).  In  addition,  the  focus  of  previous 
S  A  research  has  been  in  the  area  of  individual  versus  team  SA  (Salas,  Prince,  Baker,  &  Shrestha, 

1995) .  In  the  present  study,  we  focused  on  team  SA  as  a  process  that  is  measured  separately  from 
performance.  In  this  manner,  we  established  all  of  our  T^RM  subprocesses  as  potential  causal 
antecedents  to  mission  performance.  Important  SA  behaviors  include:  confirming  and  cross¬ 
checking  information  within  the  team,  communicating  relevant  information  to  crewmembers, 
integrating  information,  etc.  Our  results  then  tie  team  SA  to  mission  performance  and,  as  we 
detailed  previously,  also  provide  concrete  and  reinforceable  S A  behaviors. 

FA.  Past  research  has  also  blurred  the  distinction  between  workload  and  SA  (e.g.,  Hendy, 

1996) .  In  our  study,  we  assess  a  related  concept,  FA,  where  the  consequences  of  poor  FA  are 
either  work  overload  or  work  underload  for  specific  crewmembers.  We  found  that  FA  behaviors 
involving  clear  and  overt  stating  of  crewmembers’  roles  and  responsibilities  were  highly 
correlated  with  overall  mission  performance. 

TM.  In  our  study,  crews  who  stated,  monitored,  and  “stuck  to”  planned  times  performed 
better  than  those  crews  who  did  not.  The  SOF  mission  used  in  our  research  was  very  HftmanHing 
and  event-filled,  imposing  severe  time  pressures  on  all  participants.  In  fact,  many  crewmembers 
commented  that,  in  the  real  world,  they  would  plan  for  such  a  mission  over  2-3  days  rather  than 
the  3-4  hours  provided — conditions  that  could  easily  lead  to  a  need  for  good  TM. 

Moray,  Dessouky,  Kijowski,  and  Adapathya’s  (1991)  application  of  scheduling  theory  to 
the  optimization  of  a  simple  human  performance  task  perhaps  offers  some  descriptive  value  here. 
Scheduling  theory  is  primarily  used  in  industrial  engineering  to  address  problems  of  timing  and 
sequencing  of  operations.  The  questions  that  scheduling  theory  address  in  a  machine 
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environment  are  similar  to  those  a  SOF  mission  team  must  address;  In  what  order  should  tasks  be 
tackled?  Which  (if  any)  task  should  be  interrupted,  and  when?  How  long  will  the  completion  of 
all  tasks  take?  Will  there  be  any  spare  time  between  tasks,  and  how  should  it  be  used? 

Like  the  Moray  et  al.  (1991)  research,  the  SOF  mission  teams  we  observed  had  several 
tasks  to  perform  (e.g.,  AR,  AD,  prepare  charts,  etc.).  The  teams  were  free  to  choose  the  order 
and  timing  of  tasks  that  were  performed  for  MP  and  much  of  the  timing  of  events  during  mission 
execution.  Scheduling  theory  assumes  that  despite  all  these  choices  and  tasks,  explicit 
application  of  appropriate  rules  and  selection  criteria  (e.g.,  number  of  tasks  to  be  completed  or 
timelines)  will  lead  to  optimal  performance  and  reduce  subjective  workload.  Thus,  those  crews 
who  defined  crewmember  roles  and  responsibilities  (FA)  and  explicitly  stated  their  timelines  of 
performance  (TM)  were  implicitly  applying  scheduling  rules  and  selection  criteria  to  be  met. 
This  could,  in  turn,  explain  their  superior  performance  compared  to  other  crews  who  did  not 
provide  this  explicit  rule-based  direction  to  help  guide  their  collective  actions. 

TE.  The  truly  unique  area  we  applied  to  CRM  was  TE.  Although  one  can  logically  assume 
that  an  individual’s  threat  knowledge  and  understanding  will  aid  performance  in  a  hostile 
environment,  there  has  been  virtually  no  work  done  on  distribution  of  tactical  knowledge  among 
a  team  and  the  team  s  application  of  that  knowledge.  This  is  the  essence  of  our  TE  subprocess. 
We  found  that  better  TE  was  associated  with  better  mission  performance,  where  we  focused  on 
issues  like:  crewmember  exchange  of  threat  information,  threat  response  in  the  context  of  other 
mission  players,  and  discussion  of  various  tactical  choices. 

C3.  Within  the  context  of  other  CRM  research,  perhaps  our  most  interesting  finding  was 
that  C3  was  not  correlated  with  mission  performance.  A  host  of  CRM  researchers  have  used 
communication  as  their  primary  index  of  crew  coordination  (e.g.,  Kanki,  Lozito,  &  Foushee, 
1989;  Krumm  8c  Farina,  1962;  Lassiter,  Vaughn,  Smaltz,  Morgan,  &  Salas,  1990).  Interestingly,' 
most  of  these  studies  found  various  communication  correlates  with  their  performance  measures. 
For  example,  Kanki  et  al.  (1989)  foimd  that  homogeneity  of  speech  patterns  (use  of  more 
standardized  and  predictable  speech)  was  characteristic  of  crews  with  fewer  performance  errors. 
Accordingly,  Lassiter  et  al.  (1990)  rated  two-person  helicopter  teams  on  their  communication 
and  mission  performance  and  foimd  a  significant  positive  correlation. 

There  are  several  potential  reasons  for  our  null  result  in  terms  of  the  link  between 
communication  and  mission  performance.  One,  we  took  an  extremely  tactical  view  of  C3, 
defining  it  to  include  the  crew  s  use  of,  and  commumcations  with,  external  team  members  (e.g., 
coordinating  action  items  with  ground  parties)  as  well  as  the  crew  itself.  As  such,  our  C3  units  of 
measurement  were  not  at  the  level  of  detail  (e.g.,  transcribed  “thought  units”)  of  much  of  the 
past  work  on  communication  and  performance.  The  fact  that  our  C3  focus  included  outside 
parties  may  have  also  lessened  sensitivity  of  the  measure  due  to  current  operating  constraints  as 
well  as  the  use  of  a  single  simulator.  That  is,  a  key  operational  component  of  C3  for  most  SOF 
mission  teams  involves  coordinating  mission  events  with  other  mission  players,  which  our  C3 
measurement  indices  were  designed  to  assess.  These  outside  agencies  and  players  were  typically 
role-played  by  instructors  or  our  P/0  which  admittedly  has  a  certain  artificiality.  Thus,  our 
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measure  of  C3  may  be  a  more  powerful  predictor  of  performance  in  the  context  of  a  multiship 
simulator  mission  (e.g.,  DMT)  where  other  players  are  actually  present. 

Another  potential  reason  for  the  lack  of  a  C3 -performance  relationship  is  that  at  the  ART 
level  of  training,  communication  skills  and  procedures  have  already  been  ingrained  in  SOF 
aircrews  and,  as  a  result,  there  is  limited  variability  within  this  T^RM  subprocess.  Supporting 
evidence  for  this  notion  comes  from  research  done  on  the  Crewmember  Management  Attitudes 
Questionnaire  (CMAQ)  (Merrit,  1996).  In  a  cross-cultural  analysis  of  individual  CMAQ  items, 
she  found  that  those  items  relating  to  communications  were  ""universally,  negatively  skewed, 
suggesting  strong  pilot  endorsement  of  the  concept”  (p.97).  As  such,  the  communication  items 
have  limited  diagnosticity  between  crews  or  crewmembers. 

In  addition,  several  post  hoc,  SME  interviews  suggested  that  operating  requirements  that 
are  instilled  in  MC-130P  SOF  aircrews  demand  that  crew  communication  within  and  without  the 
cockpit  be  held  to  an  absolute  minimum.  One  interviewee  put  it  quite  succinctly  when  he  said, 
""shut  up  and  be  there-that  is  our  tactic.”  He  further  explained  that  the  goal  of  an  MC-130P 
cockpit  is  to  be  quiet  and  efficient  during  combat  missions.  In  fact,  one  of  the  items  that  must  be 
completed  on  the  MC-130P,  low-level  checklist  is  to  turn  off  the  wafer  switch  to  ensure  that  no 
external  transmissions  occur.  Other  SMEs  concurred,  stating  that  during  tactical  missions  one 
does  not  want  too  much  communication  in  the  cockpit  because  this  will  potentially  distract 
crewmembers  from  the  threats.  These  descriptions  of  the  MC-130P  environment  explain  the  low 
correlation  between  C3  process  ratings  and  crew  tactical  mission  performance.  Further 
endorsement  of  these  descriptions  was  found  when  we  looked  at  the  variance  of  the  rated 
subprocesses  (SA  =  13.04,  TE  =  10.83,  TM  =  9.65,  C3  =  7.89,  and  FA  =  5.78).  C3’s  relatively 
low  value  indicates  perhaps  that  there  was  not  much  for  our  researcher/observer  to  rate,  as  most 
of  the  crews  were  following  typical  ""comm  out,”  MC-130P  procedures. 

We  stress  this  divergence  from  the  literature  on  CRM  to  illustrate  the  importance  of 
specific  tactical  or  even  weapon  system-specific  CRM  training  as  opposed  to  general  CRM 
training.  That  is,  this  finding  is  atypical  of  the  work  in  commercial  aviation  which  shows  the 
relationship  of  communications  to  performance  (e.g.,  Predmore,  1991).  It  is  one  factor  (of 
perhaps  many)  that  clearly  distinguishes  a  tactical  environment  from  commercial  aviation 
environments.  It  may  be  a  subprocess  that  will  also  have  varying  relevance  across  airframes.  One 
SME  pointed  out,  for  example,  that  the  MH-53J  may  require  more  communication  and 
coordination  due  to  the  nature  and  requirements  of  their  mission  as  opposed  to  the  MC-1 30P. 

Phase-specific  performance  and  phase-specific  PRM  indices  were  positively  correlated,  but 
none  were  significant 

Although  none  of  the  phase-specific,  process-performance  relationships  were  statistically 
significant,  they  all  trended  positive  and  provide  additional  evidence  of  the  strong  relationship 
between  crew  coordination  and  mission  performance.  One  potential  reason  for  the  decreased 
process-performance  correlations  is  that  ratings  by  mission  phase  are  bracketed  in  time  where 
raters  based  their  assessments  on  smaller  samples  of  crew  behaviors.  As  such,  mission  phase 
ratings  may  not  be  as  stable  as  overall  ratings. 
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The  individual  PRM  subprocesses  had  differential  impact  on  mission  performance  across  the 
phases. 

The  differential  impact  of  T^RM  subprocesses  on  phase-specific  mission  performance  has 
important  implications  for  CRM  training  developers  and  instructors.  Specifically,  optimal  crew 
coordination  training  would  highlight  different  subprocesses  depending  on  the  mission,  phase, 
and  key  training  events.  Below,  we  offer  brief  explanations  for  the  patterns  of  subprocess 
correlations  with  performance  in  each  of  the  five  mission  phases  (Table  10). 

MP.  SA  had  the  highest  correlation  with  MP  performance,  closely  followed  by  TM.  This 
result  is  not  surprising  if  one  considers  the  tasks  of  MP  (reading  the  FRAG,  detailed  map  study, 
calculations  of  route  and  leg  times,  route  planning,  reviews  of  EW  and  Intel  information,  etc.), 
combined  with  our  definition  of  SA  as  an  “integration  of  multiple  pieces  of  information.”  The 
associated  high  correlation  with  TM  suggests  that  good  MP  will  result  when  information  is  inte¬ 
grated  in  a  timely  and  efficient  manner. 

Also  notable  was  the  low  correlation  between  FA  and  MP  performance.  It  is  striking 
because  of  the  number  of  tasks  (e.g.,  charts,  briefs,  flight  plans,  E&E  plan,  etc.)  that  must  be 
completed  during  the  compressed  MP  time  period  for  this  mission.  It  might  be  logically  assumed 
that  crews  who  distribute  the  duties  and  resources  effectively  and  efficiently  would  have  superior 
MP  performance.  The  relatively  low  correlation  between  FA  and  MP  performance  (r  =  .22), 
however,  indicates  that  this  is  not  necessarily  so.  SME  interviews  shed  light  on  this  seemingly 
perplexing  result.  An  operational  MC-130P  pilot  smiled  and  said,  “This  is  typical  of  this 
community;  we  just  let  the  navs  plan.” 

LL.  FA  had  the  highest  correlation  with  LL  performance.  Although  delimiting  crew  roles 
and  responsibilities  (FA)  seem  to  be  highly  correlated  with  all  mission  execution  phases  (see 
Table  10),  its  relative  impact  was  the  highest  for  LL.  It  is  perhaps  the  duration  and  diversity  of 
the  activities  that  occur  during  LL  which  make  FA  so  critical.  For  the  other  mission  phases 
(excluding  MP),  the  focus  was  on  a  specific  mission  event  (an  AD,  an  AR,  or  an  I/E).  However, 
there  are  multiple  mission  events  during  LL,  and  tasks  must  be  performed  simultaneously  and 
continuously  (vs.  discretely).  Therefore,  it  may  be  more  critical  to  clearly  delineate  crewmember 
functions  to  better  perform  during  LL. 

Somewhat  surprising  in  terms  of  the  subprocess  rating  correlations  with  this  mission  phase 
was  the  extremely  low  correlation  between  TE  and  LL  performance  (r  =  .06).  SMEs  pointed  out 
two  potential  reasons  for  this.  One,  they  suggested  that  artificiality  of  the  simulator  environment 
may  be  driving  TE  behavior  more  than  the  other  subprocesses.  That  is,  crews  know  that  the 
scenario  will  run  for  the  scheduled  mission  and  that  they  will  not  necessarily  suffer  any  of  the 
consequences  (e.g.,  death  or  mission  termination)  of  poor  threat  tactics.  Two,  SMEs  cited  the  fact 
that  in  the  real  world,  MC-130P  crews  do  not  routinely  fly  penetrating,  low-level  missions  in  a 
threat  environment,  and  as  such,  TE  should  not  be  expected  to  be  a  discriminating  factor. 
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AD.  TM  had  the  highest  correlation  with  AD  performance,  closely  followed  by  SA.  One  of 
the  most  critical  performance  requirements  for  an  AD  is  hitting  the  target  at  the  planned  drop 
time.  Crews  that  were  best  at  achieving  this  performance  requirement  were  those  who  called  the 
drop  warnings,  and  called  and  executed  the  associated  checklists  in  a  timely  fashion.  In  addition, 
SMEs  suggested  that  SA’s  relative  impact  on  AD  performance  was  understandable  given  the 
demands  of  the  AD.  They  noted  the  high  workload  demands  of  an  AD  posed  by  running  several 
checklists  and  reconfiguring  the  aircraft.  They  also  noted  the  importance  of  integrating  multiple 
crew  perspectives  and  information  as  the  crew  works  to  identify  the  DZ.  All  of  these  factors 
require  high  crew  SA  to  enable  superior  performance. 

AR.  For  reasons  not  immediately  obvious,  TE  had  the  highest  correlation  with  AR 
performance.  In  fact,  TE-AR  was  the  only  correlation  among  the  25  subprocess,  phase-specific 
performance  pairings  that  reached  statistical  significance  after  correcting  for  multiple 
correlations.  AR  is  one  of  the  primary  functions  of  MC-130P  crews,  and  as  a  result,  the 
crewmembers  who  arrive  at  ART  have  considerable  experience  with  this  mission  event. 
Consequently,  many  of  the  AR-specific  behaviors  that  fall  imder  our  other  T^RM  processes — 
such  as  coordination  with  the  helos  (C3),  timely  coordination  and  execution  of  checklists  (TM), 
or  establishing  the  FE’s  duties  (FA) — ^may  all  be  second  nature  to  these  crews.  The  process 
behaviors  that  discriminate  good  ARs  from  other  ARs  seem  to  be  those  of  a  tactical  nature: 
recognition  of  the  need  to  lower  the  suggested  altitude  to  better  avoid  potential  threats, 
establishing  loitering  procedures  over  water,  and  discussing  threat  countermeasures  while  on  the 
AR  track. 

I/E.  Like  MP,  I/E  requires  crews  to  integrate  various  information  sources  (the  tower,  the 
reception  party,  ABCCC  reports  on  the  threat  situation,  the  customer,  etc.).  And  like  AD,  the 
crew  must  reconfigure  the  airplane,  run  multiple  checklists,  and  incorporate  multiple 
crewmember  perspectives.  All  of  this  is  compoimded  by  having  the  pilots  land  while  wearing 
NVGs.  Not  surprisingly,  SA  had  the  highest  subprocess  correlation  with  I/E  performance,  closely 
followed  by  FA. 

Predominant  squadron  affiliation,  crew  orientation,  crew  structure,  and  crew  size  all 
potentially  influence  crew  mission  effectiveness. 

We  did  not  explicitly  manipulate  any  of  these  factors,  so  we  strongly  recommend  further 
research  to  more  definitively  establish  their  relationship  to  crew  effectiveness.  However,  the 
initial  trends  appear  to  support  the  view  that  training  is  an  organizational  episode  (Baldwin  & 
Magjuka,  1997).  The  primary  thrust  of  this  view  is  that  there  are  various  factors  outside  of  the 
delivery  of  training  itself  that  will  affect  the  efficacy  of  the  training  and  the  performance  of  the 
group,  or  in  this  case,  the  crew.  Three  influential  pretraining  factors  are:  Ae  introduction  of 
training,  the  social  cohort,  and  the  transfer  climate.  Of  particular  importance  to  the  above 
findings  is  the  influence  of  social  cohort.  Baldwin  and  Magjuka  (1997)  note  that  just  bringing 
trainees  together  does  not  necessarily  yield  effective  training.  Group  composition  factors  such  as 
individual  backgroimds,  group  size,  and  crew  orientation  will  all  affect  the  group  training 
effectiveness.  Though  our  sample  size  precluded  statistical  testing,  we  have  preliminary  evidence 
for  comparable  effects  with  aircrew  performance. 


64 


Past  CRM  training  hours  were  not  related  to  crew  mission  effectiveness. 

Conclusions  based  on  a  null  result  are  tenuous  at  best,  but  this  one  merits  reiteration — ^a 
crew’s  past  CRM  training  experience  (as  defined  by  number  of  CRM  training  hours)  was  not 
related  to  mission  effectiveness.  There  are  several  possible  reasons  for  the  lack  of  a  relationship. 
On  the  one  hand,  it  is  difficult  to  quantify  a  crew ’s  experience  in  CRM  training,  since  the  metrics 
to  combine  individual  training  hours  into  a  meaningful  group  index  are  not  yet  known.  On  the 
other  hand,  it  is  more  likely  that  given  the  tactically  demanding  events  of  this  CMT  scenario,  the 
generic  concepts  typically  covered  in  USAF  CRM  courses  would  offer  little  unique  value,  hence 
a  null  result  is  entirely  plausible.  At  least  at  the  ART  level,  CRM  training  requires  further 
refinement  to  include  more  tactically  relevant  concepts.  This  topic  will  be  discussed  in  the 
closing  subsection. 

Concrete  behaviors  depicting  the  PRM  process  and  mission  performance  of  effective  and 
ineffective  crews  can  be  isolated. 

Finally,  the  present  study  showed  convincingly  that  concrete  behaviors  depicting  the 
process  and  performance  of  effective  and  ineffective  crews  can  be  isolated.  There  is  presently  a 
movement  within  USAF  CRM  curriculum  development  toward  emphasizing  skills  and  behaviors 
versus  concepts  (Wilson,  1995).  By  establishing  the  behaviors  that  coincide  with  each  T^RM 
subprocess,  we  can  help  further  movement  in  that  direction.  As  Helmreich  and  Foushee  (1993) 
succinctly  state:  “It  can  be  argued  that  programs  that  employ  concrete  behavioral  examples 

should  have  a  greater  impact  on  crew  processes  and  outcomes  than  those  that  deal  with  abstract 
concepts  (p.  27).” 

Each  of  these  results  has  important  implications  for  training,  evaluating,  and  promoting 
CRM  concepts  among  instructors  and  students.  Some  concrete  training  recommendations  based 
on  these  central  findings  are  described  in  the  closing  subsection. 

Implications  for  CRM  Research 

The  diversity  and  richness  of  the  empirical  data  produced  by  the  present  research  suggest  a 
number  of  directions  for  conducting  follow-on  research.  In  this  subsection,  we  discuss  eight 
“thrusts”  that  we  believe  will  have  the  largest  payoff  for  creating  a  substantive  knowledge  base 
to  support  future  CRM  research. 

A  first  thrust  entails  developing  an  understanding  of  the  nature  of  our  quasi-experimental 
methodology  and  having  an  appreciation  for  why  it  was  successful.  Specifically,  the  use  of  an 
SME-researcher  to  collect  team  process  data,  following  Povenmire  et  al.  (1989),  is  one  of  the 
cornerstones  to  our  T  RM  measurement  model.  From  our  experience,  one  cannot  overstate  the 
importance  of  having  a  highly  experienced  researcher  serve  as  one  of  the  observers.  For  example, 
Nullmeyer,  Bruce,  and  Spiker  (1994)  have  noted  that  the  complexities  of  CMT,  as  with  any 
orgamzational  system,  require  the  use  of  an  SME  with  an  “eye  trained  for  spotting  relevant 
events  in  what  could  be  an  unintelligible  thicket  to  the  uninitiated.”  When  armed  with  a 


65 


structured  observational  protocol,  the  SME-researcher  gives  one  a  more  “precise  lens”  on  what 
otherwise  seems  like  a  chaotic  week  of  training,  thereby  increasing  the  chances  that  critical 
T^RM  behaviors  will  be  captured.  This  stands  in  contrast  to  videotaping,  which  can  be  costly  and 
inefficient,  and  highly  objective  checklists,  which  can  be  used  by  nonSME-observers  but  may 
not  capture  the  tactically  relevant  behaviors  specific  to  a  mission  scenario. 

Given  we  accept  the  importance  of  an  SME-observer,  we  must  identify  the  key 
qualifications  he/she  should  possess  for  use  in  future  research.  Based  on  the  present  research,  it  is 
clear  that  the  SME  should  have  expertise  in  the  tactical  domain  of  interest,  experience  in  training 
student-crews  in  the  particular  weapon  system,  and  an  affinity  for  and  appreciation  of  research. 
Such  individuals  are,  of  course,  not  easy  to  find.  However,  the  methodological  rewards  from 
obtaining  the  services  of  such  an  individual  are  certainly  worth  the  search.  As  an  added  benefit, 
materials  developed  for  use  by  the  SME-researcher  will  be,  with  only  minor  modifications, 
equally  well-suited  for  the  CMT  instructors.  Hence,  protocols  that  help  the  SME-researcher 
observe  and  capture  notable  T^RM  behaviors  can  also  be  used  by  the  CMT  instructors  to  identify 
and  reinforce  those  same  behaviors. 

A  second  methodological  thrust  concerns  the  importance  of  maintaining  independent  esti¬ 
mates  of  process  and  performance  data  in  all  future  research  endeavors.  Not  only  will  such  inde¬ 
pendence  ensure  that  one  obtains  an  unbiased  estimate  of  the  strength  of  the  relationship  between 
team  process  and  mission  performance,  it  will  help  to  ascertain  the  more  “dynamic”  crew-team 
processes  that  can  potentially  be  impacted  through  targeted  training  interventions.  As  a  follow-up 
to  the  present  study,  we  will  be  reviewing  in  detail  the  SME-generated  notes  from  the  T-MOT  to 
extract  the  most  notable  T^RM  behaviors,  positive  and  negative,  which  can  be  cost  effectively 
targeted  during  CMT. 

At  an  even  more  basic  level,  there  is  a  pressing  need  for  additional  empirical  studies  that 
confirm  a  significant  correlation  between  team  coordination  processes  and  mission  performance. 
These  are  the  correlations  that  tell  us  which  processes  are  most  important,  i.e.,  have  the  largest 
impact  on  performance.  In  this  regard,  it  is  telling  that  there  have  been  so  few  published  attempts 
at  replicating  the  basic  Povenmire  et  al.  (1989)  process-performance  correlation  in  the  tactical 
domain.  On  the  one  hand,  such  replications  are  difficult  since  they  require  on-site  observers 
rather  than  recording  audio  communications  or  videotaping  student  crews.  Training  resources  are 
limited,  so  it  is  imperative  that  we  have  the  means  to  unequivocally  identify  high  payoff 
processes,  and  then  develop  and  target  interventions  to  train,  reinforce,  and  sustain  them.  At  a 
minimum,  this  will  mean  that  subsequent  studies  have  two  researcher/observers,  or  at  the  very 
least,  one  process-observer  and  some  independent  means  (e.g.,  via  computer)  of  recording 
mission  performance.  In  the  context  of  our  T^RM  measurement  model  (see  Figure  1),  we  have 
already  expressed  a  belief  that  performance  will  need  to  be  handled  almost  as  delicately  as 
process,  by  means  of  an  observer  who  has  access  to  the  full  range  of  materials  created  during 
CMT  (annotated  charts,  ground  tracks,  etc.). 

Third,  it  will  be  important  to  show  that  the  present  study’s  observation  of  a  pronounced 
relationship  between  process  and  performance  is  not  limited  to  a  particular  weapon  system, 
mission  scenario,  or  training  regime.  Indeed,  the  generalizability  of  any  finding  is  traditionally 
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considered  to  be  a  logical  next  step  for  research.  While  not  always  viewed  as  theoretically 
exciting,  replication  of  basic  phenomena  in  other  contexts  is  essential  to  advancement  of  any 
research  base.  Importantly,  replication  of  a  study’s  principal  finding  is  a  more  convincing  form 
of  scientific  evidence  than  is  the  size  of  the  statistical  finding  in  the  original  study  (Cohen,  1994). 

For  example,  crew  complement  in  the  MC-130P  WST--six--is  rather  large.  Do  team 
process  effects  still  exist  when  crew  size  is  reduced  by  half-to  three-as  in  the  MH-53J  WST? 
Does  the  nature  of  fixed-wing  operations  lend  itself  to  larger  team  process  effects  than  might  be 
foimd  with  helicopter  operations?  Future  work  is  certainly  needed  to  delineate  the  boimdaries  of 
the  size  of  T^RM  process  effects  that  one  can  expect  to  find  in  these  other  contexts.  At  one  level, 
there  is  no  a  priori  reason  why  our  present  results  would  be  characterized  by  an  unusually  large 
process-performance  association.  In  fact,  this  relationship  might  be  even  stronger  when  crew  size 
is  smaller  because  each  crewmember’s  role  is  proportionally  more  vital.  Moreover,  one  can  ask 
whether  a  large  effect  of  T^RM  process  will  still  be  observed  when  we  study  less  experienced 
crews,  such  as  the  beginning  student-crews  trained  during  MQ.  From  a  training  standpoint,  it  is 
noteworthy  that  a  substantial  process-performance  relationship  was  observed  with  the  highly 
experienced  ART  crews  of  the  present  study,  as  it  implies  that,  regardless  of  experience  level, 
there  is  room  for  improvement  via  targeted  training. 

Fourth,  the  present  results  strongly  underscore  MP  as  a  very  promising  area  for  further 
research.  On  the  one  hand,  MP  is  the  initial  phase  of  the  Day  5  ART  mission,  so  we  inevitably 
see  the  team  process  effects  first  appear.  Recall  that  in  our  assessments  of  MP  effects  on  T^RM, 
we  identified  three  types  of  crew  orientation — emission,  task,  or  training.  In  the  future,  we  will 
want  to  develop  targeted  interventions  to  promote  constructive  team-building  behaviors  during 
MP,  so  that  either  a  mission  or  task  orientation  is  adopted  by  all  crews. 

Moreover,  with  regard  to  MP,  it  is  clear  that  we  have  only  begun  to  “scratch  the  surface” 
of  this  potentially  important  area.  For  example,  does  having  access  to  an  automated  planning 
system  stifle  development  of  critical  T^RM  behaviors?  Or,  does  such  access  give  plaimers  more 
time  for  information-sharing  and  assumption-testing  activities  that  are  so  essential  for  good 
preparation  (Spiker  &  Nullmeyer,  1995b)?  Similarly,  does  having  access  to  high  fidelity 
simulation  and  geospecific  imagery  overemphasize  the  technological  aspects  of  MP  at  the 
expense  of  the  more  personal,  T^RM-oriented  issues?  These  and  other  questions  merit  study  in 
their  own  right,  particularly  as  the  SOF  community  moves  ever  closer  to  integrating 
computerized  systems  into  its  MP  process,  both  real  world  and  training. 

As  a  fifth  thrust,  future  work  should  explore  the  extent  to  which  T^RM  and  CRM  studies 
represent  two  distinct  research  strands.  That  is,  if  tactics  play  a  dominant  role,  does  that  mean 
that  CRM  principles  identified  from  the  commercial  airlines  industry  no  longer  apply?  At  the 
present  time,  we  do  not  know  the  answer  to  this  question,  although  the  present  study  provides 
some  preliminary  data  to  consider.  Specifically,  TE  and  SA  comprise  the  two  subprocesses  that 
are  more  scenario-dependent  than  are  FA,  TM,  and  C3.  From  reviewing  the  content  items  of  the 
T-MOT,  it  is  clear  that  both  subprocesses  are  marked  by  notable  behaviors  that  vary  considerably 
across  mission  phase.  For  example,  the  TE  subprocess  is  represented  by  such  diverse  behaviors 
as  use  of  terrain  and  altitude  (LL),  technical  proficiency  (AR  and  AD),  and  coordination  of  threat 
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procedures  (I/E).  On  the  other  hand,  a  subprocess  such  as  FA  is  characterized  by  such  stable 
behaviors  as  “  crew  involvement”  that  change  little  from  phase  to  phase. 

To  assess  the  phase-specific  nature  of  these  subprocesses,  we  computed  the  overall 
between-crew  variability  in  T-MOT  ratings  as  well  as  the  between-crew  variability  within  each 
phase.  As  expected,  overall  standard  deviations  for  the  ratings  were  higher  for  SA  (3.61)  and  TE 
(3.29)  compared  to  the  other  three  subprocesses  (2.40-3.11).  In  addition,  when  standard 
deviations  were  computed  for  each  phase  separately,  TE  and  SA  had  either  the  first  or  second 
highest  standard  deviation  within  each  of  the  five  mission  phases.  Taken  as  a  whole,  these 
analyses  suggest  that  (a)  tactical  behaviors  play  a  major  part  of  a  crew’s  T^RM  and  (b)  these 
behaviors  will  likely  be  more  variable  in  their  definition  and  occurrence  across  missions,  weapon 
systems,  scenarios,  and  crews.  Given  such  likely  variability  in  a  major  component  of  T^RM,  it 
stands  to  reason  that,  at  least  within  the  CMT  domain,  CRM  courses  should  be  weapon  system- 
specific. 

A  sixth  thrust  concerns  the  need  to  have  future  research  take  a  closer  look  at  T^RM 
processes  associated  with  teams  for  which  a  variety  of  non-aircrewmembers  (e.g.,  Intel)  interact 
with  the  aircrew.  The  CMT  environment  of  the  present  research  effort  only  partially  captured  the 
flow  of  information  in  a  larger  team  through  the  use  of  role-playing  instructors  during  MP  and 
mission  execution.  Yet  we  know  that  such  information  exchange  plays  a  major  role  in  the  crew’s 
decision  making  process,  and  hence  is  very  relevant  to  T^RM  principles.  To  gain  a  broader 
perspective  on  “teamness,”  it  is  necessary  to  apply  our  team  measurement  methodology  to  a 
larger  tactical  environment  in  which  more  individuals  play  an  active  role  in  the  mission.  Possible 
environments  for  supporting  a  more  team-intensive  scenario  include  the  Day  4  ART  networked 
training  mission,  Joint  Readiness  Training,  DIS  demonstrations,  and  joint  field  exercises. 

It  is  our  firm  belief  that  the  basic  methods  of  team  measurement  would  still  apply  in  these 
other  environments,  although  additional  T^RM  subprocesses  may  be  identified  and  operationally 
defined.  As  before,  a  front-end  analysis  would  be  needed  to  identify  these  new  subprocesses  and 
determine  how  they  relate  to  the  existing,  crew-based  ones.  Many  other  candidates  will  emerge, 
but  several  possibilities  for  new  T^RM  subprocesses  include  decision  making,  workload  distribu¬ 
tion,  and  “team”  awareness  (i.e.,  understanding  one’s  crew  role  within  the  larger  team 
objective).  Implications  for  conducting  research  within  this  “larger  team  laboratory”  will  be 
further  elaborated  in  the  next  subsection. 

A  seventh  research  thrust  would  entail  exploring  the  effects  of  individual  crewmember 
positions  on  T^RM  processes  and  behaviors.  In  this  regard,  we  primarily  emphasized  T^RM 
subprocesses  and  behaviors  at  the  crew  level.  However,  our  data  collection  instruments  (e.g.,  the 
T-MOT)  were  designed  to  also  capture  T^RM-related  activities  at  the  individual  level.  In  a 
follow-up  to  the  present  study,  we  are  analyzing  the  existing  data  set  to  determine  if  there  are 
pronounced  effects  of  T^RM  associated  wOith  the  characteristics  and  proficiencies  of  individual 
crewmembers  (Silverman,  et  al.,  in  press). 

While  definitive  answers  to  the  above  questions  await  further  research,  we  looked  at  the 
latter  question  by  computing  the  correlations  between  the  T-MOT-based  T^RM  ratings  associated 
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with  each  crew  position  and  the  crew’s  overall  T^RM  rating.  Results  showed  a  wdde  range  of 
correlations,  indicating  that  a  crew’s  overall  T^RM  is  not  simply  an  arithmetic  average  of  the 
individual  crewmember  ratings.  Specifically,  we  observed  the  following  correlations:  LN  (.89), 
CP  (.86),  AC  (.79),  RN  (.64),  FE  (.41),  and  CSO  (.20).  This  pattern  is  quite  instructive,  as  it 
gives  us  some  insight  into  crewmember  influences  on  the  T^RM  process,  at  least  for  the  MC- 
130P  weapon  system. 

The  LN’s  large  impact  on  crew  T^RM  is  consistent  with  his/her  role  as  developer  of  the 
flight  plan  during  MP  and  continual  guidance  he/she  provides  to  the  pilots  throughout  the 
mission.  The  high  correlation  for  the  CP  might  be  considered  surprising,  although  it  is  our 
opinion  that  it  reflects  the  fact  that  the  more  effective  crews  made  better  use  of  the  CP’s  time, 
particularly  during  MP,  which  translated  into  more  effective  T^RM  behaviors  during  the  mission. 
On  the  other  hand,  the  low  correlations  for  the  FE  and  CSO  are  consistent  with  our  observations 
that  both  crewmembers  were  underutilized  during  much  of  the  MP  process,  and  were  similarly 
under-engaged  during  selected  mission  phases.  It  vidll  be  interesting  to  determine  whether  a 
comparable  pattern  of  correlations  holds  up  under  other  mission  conditions,  and  whether  other 
weapon  systems  exhibit  a  similar  wide  range  of  crewmember  T^RM  associations. 

An  eighth  and  final  thrust  involves  the  identification  of  critical,  stable,  and  effective  crew 
process-behaviors  that  will  form  the  core  of  a  T^RM  content  domain,  regardless  of  the  tactical 
scenario,  mission,  or  weapon  system.  At  the  present  time,  we  do  not  know  what  these  founda¬ 
tional  behaviors  are.  The  five  T^RM  subprocesses— and  associated  T-MOT  behaviors— 
addressed  in  the  present  study  are  only  a  first  attempt  at  establishing  their  identity.  The  detailed 
analysis  of  T^RM  behaviors  during  the  MP  phase  gave  us  some  further  insights,  but  additional 
research  in  this  area  is  clearly  needed. 

To  illustrate,  in  reviewing  the  T-MOT  data  from  the  three  most  successful  crews,  the 
follovdng  observations  were  commonly  made.  First,  the  AC  provided  verbal  backups  during 
periods  of  communication  ambiguity  and  kept  the  crew  informed  (as  required)  on  what  was 
happening.  The  LN  integrated  inputs  from  the  RN  and  CP,  and  helped  the  AC  make  decisions. 
The  other  crewmembers  accepted  AC  and  LN’s  communication  as  primary,  and  theirs  as  secon¬ 
dary.  Moreover,  multiple  crew  positions  caught  and  corrected  errors  by  other  crewmembers 
when  they  got  behind  or  tired  (e.g.,  the  AC  and  RN  catch  and  correct  navigation  errors  by  the 
LN).  The  CSO  “passed  comm”  in  short  bursts  of  imderstandable  information.  As  well,  the  crew 
always  had  a  backup  plan  for  who  would  take  over  a  system  function  (e.g.,  expendable 
deployment)  when  it  came  out  of  auto  mode.  Smooth,  crisp  communication  between  the  crucial 
dyads  (AC  and  CP,  AC  and  LN,  LN  and  CP)  was  evident  throughout  the  mission. 
Coimnumcation  filtering  between  crewmembers  was  based  on  perceived  workload  and  timing  of 
the  mission  event. 

Identification  of  these  T^RM  behaviors  will  clearly  arise  on  an  incremental  basis,  but  once 
specified,  they  would  provide  the  core  content  for  a  T^RM  course  that  would  be  taught  at  the  MQ 
level.  To  make  this  determination,  we  would  first  look  for  T^RM  behaviors  that  appear  to 
transcend  tactical  mission  phases  and  scenarios.  Then,  we  would  more  closely  examine 
individual,  weapon  system-specific  T^RM  behaviors  to  see  if  we  can  identify  antecedent  factors 
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that  tend  to  promote/reinforce  some  behaviors  as  well  as  punish  others.  Once  these  factors  have 
been  identified,  and  psychometrically  robust  instruments  developed  to  measure  their  occurrence, 
we  will  then  have  taken  a  major  step  forward  in  our  quest  to  design  training  interventions  that 
promote,  enhance,  reinforce,  and  sustain  T^RM  behaviors. 

Applicability  to  Air  Force  Combat  Team  Training  Initiatives 

There  has  been  increasing  recognition  in  recent  years  of  the  need  to  train  combat  teams. 
This  has  led  to  initiatives  that  go  beyond  traditional  crew-level  CRM  instruction  and  provide 
training  environments  for  larger  tactical  teams.  For  example,  DMT  will  provide  networked 
simulation  for  tactically  relevant  multiship  operations  (Carroll  &  Andrews,  1997).  Another 
example  is  Blue  Flag,  where  an  Air  Operations  Center  (AOC)  is  “stood  up”  to  exercise 
command  and  staff  functions  associated  with  deliberate  and  contingency  planning.  In  this 
section,  we  consider  the  applicability  of  our  methods  and  findings  to  planned  Air  Force  Research 
Laboratory,  Warfighter  Training  Research  Division  (AFRL/HEA)  research  for  these  two 
applications. 

DMT 

The  goal  of  DMT  is  to  develop  fully  integrated,  squadron-level,  ground-based 
environments  to  train  multiple  aviators  at  multiple  sites.  The  size  of  the  team  would  range  from 
individual  and  four-ship  participation  within  a  squadron  up  to  full  theater-level  battles.  Low-cost, 
high-fidelity  simulators  with  full  visual  systems  will  immerse  aircrews  in  a  training  arena,  or 
“joint  synthetic  battlespace,”  with  other  air,  ground,  sea,  and  space  forces  to  execute  the  air 
tasking  order  in  a  scenario  developed  and  managed  by  the  respective  battle  staffs  (Carroll  & 
Andrews,  1997).  Several  aspects  of  our  T^RM  measurement  model  (Figure  1)  appear  to  be 
germane  to  the  Laboratory’s  DMT  behavioral  research  program. 

We  view  T^RM  performance  as  being  multidimensional,  and  as  a  consequence,  purposely 
developed  a  data  collection  scheme  that  attended  to  multiple  facets  of  team  coordination  rather 
than  focusing  on  one  specific  dimension.  Our  finding  that  four  resource  management 
subprocesses  (SA,  TE,  FA,  and  TM)  were  reliably  correlated  with  mission  performance  for  MC- 
130P  crews  supports  this  view.  We  expect  that  performance  in  fighter  aircraft  would  be  similar. 
This  multidimensional  approach  allowed  us  to  be  very  specific  about  behaviors  to  monitor  while 
maintaining  a  reasonable  breadth  of  coverage.  Although  we  expect  our  multidimensional  view  of 
team  processes  to  generalize  across  team  performance  domains,  we  also  expect  the  specific 
categories  that  constitute  the  most  important  behavioral  dimensions  to  vary  across  aircraft  types, 
missions,  and  even  mission  elements. 

In  a  similar  vein,  we  found  that  a  combination  of  data  sources  was  needed  to  delineate 
important  T^RM  behaviors.  Our  rating  data  were  amenable  to  statistical  analyses  that  allowed  us 
to  identify  those  resource  management  areas  that  appeared  to  be  reliable  predictors  of  mission 
performance.  While  statistical  analyses  provided  a  good  filter  for  identifying  important  aspects  of 
team  performance,  the  correlations  were  not  particularly  useful  for  specifying  how  training 
content  or  strategies  might  be  improved.  Recommendations  for  training  improvement  were 
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highly  dependent  on  notes  that  accompanied  our  ratings  describing  training  treatments  (including 
feedback  or  the  lack  thereof)  and  specific  behaviors  associated  with  particularly  strong  or  weak 
crew  performance.  A  second  critical  source  of  vmderstanding  was  interviews  with  instructors  to 
interpret  and  partially  validate  the  statistical  findings. 

Another  important  aspect  of  the  present  experimental  approach,  which  we  believe  was  a 
critical  factor  in  finding  statistically  significant  correlations  between  process  and  mission 
performance,  was  attending  to  team  coordination  processes,  resulting  mission  performance,  and 
eventual  mission  outcome  (see  Figure  1).  The  intervening  mission  performance  module  reflects 
the  immediate  consequences  of  the  selected  subprocesses  during  specific  mission  phases.  The 
MC-130P  crews  were  operating  in  an  adaptive  environment,  both  with  respect  to  their  ability  to 
correct  and  modify  behaviors  in  response  to  scenario  events,  and  with  respect  to  instructor 
modifications  of  the  scenario  to  tailor  difficulty  to  the  capabilities  of  each  crew.  The  intervening 
category  of  mission  performance  provides  data  that  is  essential  if  team  coordination  behaviors 
are  to  be  linked  with  mission  outcome  measures  such  as  airdrop  accuracy  or  exposure  to  threats. 
This  module  helps  control  for  external  factors  variable  enemy  actions,  equipment  malfunctions, 
weather,  and  luck  that  could  otherwise  obscure  the  effects  of  varying  degrees  of  team 
coordination. 

The  hierarchical  structure  underlying  our  data  set  may  also  have  merit  in  a  DMT 
environment.  There  are  two  basic  strategies  that  could  be  followed  in  measuring  team 
performance.  One  is  to  identify  and  track  large-scale,  global  variables  that  have  broad  appeal,  but 
lack  specificity  for  immediate  implementation  in  CMT.  The  alternative  is  to  identify  and  track 
particular  context-specific  behaviors  that  are  more  amenable  to  immediate  training  and 
reinforcement  but  which  may  have  more  limited  generalization.  Clearly,  each  approach  has  its 
strengths  and  weaknesses.  Our  data  structure  was  designed  to  benefit  from  the  best  of  both 
strategies  by  having  an  overall  process  rating  and  a  total  performance  metric,  but  having  each 
composed  of  more  detailed  data  categories  and  elements  that  would  enable  a  more  detailed  look 
at  higher  order  relationships  that  achieved  statistical  significance.  Once  an  overall  T^RM  and 
team  mission  performance  relationship  was  established,  this  structure  allowed  us  to  further 
specify  which  mission  elements  and  which  T^RM  subprocesses  appeared  to  contribute  to  the 
effect. 

The  implications  of  the  present  research  for  the  DMT  program  are  at  least  twofold.  First,  a 
guiding  assumption  should  be  that  DMT  technology  can  support  multiple,  specific  training  goals. 
Substantial  attention  has  been  paid  to  using  DMT  for  combat  SA  training.  This  is  consistent  with 
a  portion  of  our  findings.  However,  SA  may  be  only  one  of  many  areas  that  can  be  effectively 
taught  using  DMT  if  accompanying  training  practices  are  properly  designed  and  executed. 
Second,  performance  measurement  for  both  trainee  feedback  and  training  effectiveness 
assessment  needs  to  reflect  the  multidimensional  nature  of  tactical  team  performance  and  the 
diverse  needs  of  various  users  of  the  resulting  data 
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Blue  Flag 


A  recent  Blue  Flag  exercise  provided  an  opportunity  to  overlay  the  five  T^RM  subprocesses 
from  the  MC-130P  study  on  a  substantially  different  tactical  team  operation.  Blue  Flag  involves 
standing  up  an  AOC  for  three  days  to  provide  training  on  Air  Tasking  Order  (ATO)  generation. 
More  specifically,  the  Night  Guidance,  Apportioiunent,  and  Targeting  (NGAT)  Cell  was 
observed  during  Blue  Flag  97-1.  The  NGAT  is  made  up  of  an  NGAT  cell  chief,  a  support  group 
of  about  a  half  dozen  Intel  personnel,  and  nine  specialized  working  groups  (Navy  aviation, 
strategic/attack  assets,  CAS,  tanker  support,  and  so  forth)  for  a  total  of  about  60  participants.  As 
summarized  below,  the  five  T^RM  categories  provided  a  useful  template  for  organizing 
observations,  and  behaviors  associated  with  these  five  categories  accounted  for  much  of  the 
observable  “learning  curve”  exhibited  by  Blue  Flag  participants. 

SA  -  Approximately  90%  of  the  participants  had  no  AOC  experience  in  either  an  exercise 
or  actual  combat  operations.  Although  a  basic  process  was  briefed  to  the  NGAT  cell  participants 
at  the  beginning  of  the  exercise,  the  inexperienced  participants  expended  a  great  deal  of  effort 
and  time  in  the  first  night  translating  the  conceptual  process  that  was  briefed  into  the  reality  of 
the  AOC.  Activities  included  determming  what  information  was  available  pertaining  to  their 
function,  where  that  information  could  be  obtained,  where  their  outputs  needed  to  go,  and  how 
these  products  needed  to  be  formatted.  This  appears  to  correspond  to  Endsley’s  (1995a)  Level  1 
SA.  Processing  input  data  to  generate  required  products  could  correspond  to  her  Level  2  SA. 
However,  participants  in  this  exercise  were  experienced  warriors,  and  this  level  of  processing 
appeared  to  fall  quickly  into  place  once  the  “big  picture”  of  AOC  operations  was  developed. 
Finally,  over  the  course  of  the  exercise,  daily  feedback  concerning  the  previous  day’s  products 
could  be  used  by  participants  to  generate  a  mental  model  of  the  consequences  of  their  behavior  to 
enable  a  better  match  between  effort  expended  and  likely  benefits,  corresponding  to  Endsley’s 
Level  3  SA. 

FA  -  An  obvious  change  over  the  course  of  the  exercise  was  how  initially  inexperienced 
working  group  leaders  assigned  work  in  their  particular  specialty  group.  A  typical  first  night  allo¬ 
cation  of  effort  was  for  the  group  leader  to  allocate  nothing  because  he  had  insufficient 
knowledge  of  the  task  or  required  products  to  assign  elements  to  others.  The  obvious  results  were 
late  products  and  fnistrated  team  members.  By  the  second  night,  allocation  of  tasks  had  vastly 
improved,  with  group  leaders  typically  taking  a  coordination  role  while  group  members  did  the 
required  labor-intensive  information  transformations  and  paperwork  completion. 


TE  -  The  wing  or  squadron  develops  tactics  per  se  upon  receipt  of  the  ATO.  However,  the 
NGAT  participants  who  are  responsible  for  developing  elements  of  the  ATO  need  to  develop 
their  parts  in  a  way  that  a  sound,  tactical  solution  can  be  generated  by  the  receiving  organization. 
To  assess  this  in  Blue  Flag  exercises,  selected  operational  units  receive  the  resulting  ATO  and 
provide  feedback  that  includes  the  tactical  feasibility  of  their  portions  of  the  tasking. 

TM  -  The  NGAT  process  ends  with  a  briefing  to  the  Air  Component  Commander  (in  this 
case,  a  Lt  General)  at  a  prespecified  time.  The  NGAT  cell  leader  has  a  clear  requirement  to 
monitor  progress  in  earlier  steps  of  the  process  to  accommodate  the  time  needed  to  complete  the 
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later  steps  within  this  finite  time  window.  TM  was  given  much  more  attention  by  the  NGAT 
leader  on  Night  2  than  it  was  on  Night  1 . 


C3  -  Communication  among  the  working  groups  and  between  the  working  group  and  the 
NGAT  leader  forms  the  essence  of  the  NGAT  process.  The  NGAT  chief  for  this  exercise  had 
developed  checklists  that  specified  coordination  requirements  and  provided  a  place  for  initials  to 
indicate  that  the  required  coordination  had  occurred. 

Recommendations  for  Combat  Mission  Training 

Data  from  the  present  study  (particularly  instructor  and  crew  comments)  imply  that  changes 
are  needed  in  the  CRM  training  that  is  conducted  at  the  58  TRSS.  In  this  section,  we  discuss  five 
areas  from  our  research  that  we  believe  have  implications  for  improving  CRM  training  in  both 
MQ  and  ART. 

The  first  area  pertains  to  generating  a  convincing  position  that  the  Formal  School  can 
better  serve  the  SOF  community  through  improved  CRM  or  tactical  team  training.  The  present 
ART  cumculum  has  evolved  from  a  systems  and  EP  training  focus  to  the  current  tactical  combat 
skills  training  emphasis.  This  evolution  occurred  based  on  requests  made  by  crews  during  the 
Desert  Storm  period.  In  that  time  frame,  crews  were  being  regularly  tasked  to  fly  in  operational 
missions  over  the  Gulf,  and  they  were  reporting  a  high  degree  of  proficiency  in  (basic)  systems 
and  EPs  knowledge.  However,  they  were  also  reporting  a  great  need  for  focused  tactics  and 
threat  traimng.  These  highly  experienced  crews  recognized  that  if  they  were  going  to  be  required 
to  rotate  though  a  full  week  of  ART,  they  should  be  receiving  the  training  content  they  needed 
These  same  crews  also  realized  that  the  hallmark  of  the  58  TRSS,  as  a  training  system,  was  the 
ability  to  provide  a  tailored  response  to  training  needs.  Thus,  the  ART  curriculum  was  modified 
to  accommodate  this  shift  in  stated  needs,  and  embedded  in  this  modified  training  practice  was 
an  expectation  that  an  intense,  combat  mission  scenario  in  the  simulator  was  a  good  place  to 
practice  CRM  skills  and  behaviors.  However,  the  CRM  scenario  that  we  studied  was  developed 
without  accompanying  CRM  training  objectives. 

In  MQ,  CRM  is  taught  in  academics  to  all  MC-130P  crewmembers  using  the  generic 
AFSOC  CRM  workbook.  Taking  a  “  one-size-fits-all”  approach  to  CRM  training  AFSOC 
gmdance  does  not  address  use  of  simulation  for  CRM  training  since  some  weapon  systems  do 
not  have  simulators  for  either  MQ  or  ART-leaving  it  up  to  the  Formal  School  to  decide  whether 
CRM  training  is  a  good  use  of  scarce  and  expensive  simulator  resources.  The  AFSOC  approach 
also  does  not  address  aircraft-  or  mission-specific  requirements.  Despite  several  attempts  to 
demonstrate  the  effectiveness  of  generic  CRM  training,  there  is  no  data  to  our  knowledge  to 
support  the  value  of  this  approach. 

Several  lines  of  evidence  suggest  that  there  is  room  for  improvement  in  CRM  training. 
First,  our  high  correlations  were  possible  only  because  we  found  a  wide  range  of  CRM  behaviors 
and  a  wide  range  of  mission  performance  levels  across  mission-qualified  crews.  Importantly,  the 
differences  between  the  extremes  appear  to  be  operationally  meaningful.  Second,  instructor 
comments  suggest  that  the  lack  of  solid  CRM  skills  complicates  life  on  the  flightline.  We  have 
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not  yet  delved  into  training  records  to  build  a  better  case,  but  that  would  be  a  logical  next  step. 
Third,  as  discussed  in  the  first  section,  forward-leaning  practitioners  of  CRM  training  who  are 
held  up  as  leaders  in  this  business  have  discarded  practices  similar  to  ours  in  the  areas  of 
operational  relevance  and  tailored  training. 

A  critical  activity  is  to  identify  specific  problems  that  improved  CRM  training  might 
impact  either  at  the  550th  SOS  or  in  follow-on  operational  units.  A  focused  problem  analysis  at 
the  550th  SOS  would  be  fairly  straightforward.  In  the  dynamic  environment  of  the  operational 
squadrons,  we  had  proposed  a  mission  information  collection,  analysis,  and  reporting  system 
(MICARS)  that  would  be  an  elegant  solution  to  tracking  drifting  squadron  needs  (Spiker  & 
Walls,  1995). 

The  second  area  pertains  to  establishing  training  objectives  and  associated  behavioral 
criteria  or  requirements.  These  objectives  should  be  platform-,  mission-,  and  task-specific,  and 
should  include  s|)ecifications  for  (a)  which  crewmembers  are  to  perform  which  tasks,  (b)  the 
criterion  level  for  task  performance,  and  (c)  the  scope  of  the  objective  (e.g.,  does  the  objective 
apply  to  ART  only,  or  does  it  cover  MQ  as  well?).  For  example,  both  our  process  and 
performance  data  and  SME  comments  reveal  that  mission  teams  demonstrate  a  high  degree  of 
variability  in  team  coordination  skill  and  mission  performance  across  the  observed  mission 
phases  (i.e.,  MP,  LL,  AR,  AD,  and  I/E).  This  data  also  indicate  that  a  crew’s  level  of  successful 
mission  execution  performance  is  a  function  of  the  scripted  mission  scenario’s  level  of  challenge. 
Additionally,  crew  performance  tends  to  improve  when  they  are  permitted  to  operate  in  a  low- 
threat,  minimum-difficulty  mission  environment.  Given  a  marginally  higher  threat  environment, 
however,  crew  coordination  and  mission  operations  performance  (e.g.,  chaff  and  flare 
expendables  discipline)  shows  a  definite  downward  trend.  If  criterion-based  objective  standards 
for  performance  existed,  we  could  more  reasonably  state,  in  simple  terms,  the  underlying  reasons 
for  a  crew’s  poor  performance. 

The  level  of  mission  challenge  is  a  direct  function  of  how  simulator  instructors  manipulate 
the  scripted  and  unscripted  threat  environment.  We  have  also  observed  a  wide  range  of 
variability  in  scenario  manipulation  that  is  provided  by  instructor  crews,  and  our  data  suggest 
evidence  of  disparate  levels  of  simulator  systems  and/or  Integrated  Electric  Combat  Simulation 
System  knowledge.  Changing  threat  system  variables  and/or  scenario  manipulation  has  been 
reported  as  a  difficult,  time-consuming  process,  and  oftentimes  the  cause  of  simulator 
malfunction.  Thus,  instructors  typically  avoid  scenario  script  manipulation  resulting  in 
artificially  low  levels  of  task  difficulty  and  corresponding  unmotivated  student  response.  In  this 
context,  the  Laboratory  is  working  with  the  58  TRSS  on  a  project  to  improve  the  user  interface  of 
the  lOS  to  lower  the  difficulty  of  manipulating  selected  scenario  properties  and  relieve  some  of 
this  problem.  Another  facet  of  the  problem  is  the  lack  of  standardized  training  objectives  and 
methods  that  promote  high-performance  learning  to  desired  criterion  levels. 

The  third  area  is  the  need  for  CRM  training  methods  to  satisfy  training  objectives.  We 
have  noted  that  crews  perceive  a  lack  of  relevance  to  their  CRM  training  On  the  first  day  of 
ART,  a  short  overview  of  the  “CRM  Pyramid”  (Wilson,  1995)  is  presented.  Following  this,  a 
dialogue  period  is  provided  for  volunteers  to  identify  any  related  problems,  or  “war  stories,”  that 
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may  have  been  recently  experienced.  Specifically,  ART  crewmembers  are  solicited  to  report,  on 
a  nonattribution  basis,  any  personal  or  other  knowledge  of  recent  violations  of  any  level  in  the 
CRM  pyramid  of  principles.  This  may  include  certain  crewmember  actions,  systemic  issues,  or 
even  organizational  problems.  While  the  intent  is  admirable,  the  resultant  exchange  is  usually 
less  than  desired.  Since  crews  generally  arrive  from  the  same  organizations,  it  should  not  be 
surprising  that  students  are  reluctant  to  (publicly)  divulge  information  that  may  incriminate 
others  with  whom  they  have  to  regularly  fly,  or  to  whom  they  are  required  to  report. 

Following  this  academic  training  period,  students  are  not  required  to  perform  any  CRM- 
related  skills  and  behaviors  until  the  CMT  scenario  provided  on  the  last  day  of  ART  (and  then 
with  unknown  performance  objectives  and  criteria,  as  previously  discussed).  Our  observational 
data  support  changing  the  overall  CRM  training  method  to  achieve  specific  team  coordination 
training  objectives,  and  provide  an  emphasis  on  CRM  skill  performance  to  meet  certain  criterion 
levels.  We  envision  a  revamped  CRM  curriculum  that  includes,  for  example:  academic  training 
on  the  emphasis  of  complete  use  of  all  knowledge  and  personnel  resources  (the  CSO,  especially); 
a  re-thinking  of  traditional  crew  position  roles  and/or  responsibilities;  a  tailoring  of  the  training 
matter  to  include  weapons-specific  and  tactically  relevant,  topical  matter;  and  an  increased 
coordination  of  high-intensity  threat  and  mission  tactics.  Finally,  a  modular  approach  to 
simulator  utilization,  and  a  flexible  approach  to  CMT  scenario  construction,  should  be  employed. 

The  fourth  area  concerns  feedback  both  to  student  crews  concerning  CRM  performance 
and  to  the  training  organization  concerning  the  effectiveness  of  the  training  itself  With  respect  to 
student  feedback,  we  observed  that,  while  there  was  a  wide  range  in  how  well  the  crews 
performed  in  our  study,  there  was  unfortunately  little  difference  in  how  the  crews  were  debriefed 
by  the  instructors.  Crews  were  given  very  little  explicit  feedback  concerning  deficiencies  in  then- 
crew  coordination  processes,  either  during  mission  planning  or  while  they  were  performing  the 
simulator  mission.  A  valuable  opportunity  to  correct  deficient  behaviors  and  reinforce  effective 
behaviors  is  therefore  lost.  Additionally,  instructors  are  demonstrating  a  certain  acceptance  of 
traditional  student  critique  methods  that  are  focused  only  on  aspects  of  procedural  performance. 


The  problem  has  several  facets.  The  first  is  that  there  is  no  consensus  that  CRM  is,  in 
reality,  a  training  objective  of  the  scenario.  In  fact,  there  is  some  feeling  among  the  contractor 
instructors  that  the  USAF  does  not  actively  view  CRM  as  an  appropriate  feedback  topic  area 
because  it  falls  into  the  area  of  techmque.  Another  problem  is  that  no  method  exists  for 
instructors  to  provide  students  with  comprehensive  individual  or  crew  assessments  of  mission  or 
crew  coordination  performance.  The  current  method  of  critique  and  feedback  reduces  the  tactical 
scenario  to  an  individual,  part-task  training  level,  and  does  not  address  the  individual  or  crew 
aspects  of  team  coordination  and  mission  performance.  Informal  discussions  with  several  CRM 
trainers  revealed  that  lack  of  meaningful  feedback  precludes,  for  the  most  part,  any  beneficial 
impact  of  the  scenario  on  future  student  performance. 

Students  need  to  be  informed  when  their  coordination  performance  is  poor  or  substandard, 
otherwise  training  cannot  hope  to  change  inadequate  or  inappropriate  behavior.  We  acknowledge 
that  instructors  are  capable  of  providing  this  feedback,  but  they  are  not  afforded  either  the 
direction  (or  even  permission)  or  the  tools  to  provide  a  focused,  or  memorable  structure  in  their 
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debrief.  We  would  advise  development  of  an  instructor’s  debrief  tool  (probably  in  checklist 
format)  that  would  enable  instructors  to  use  the  debrief  period  as  an  opportunity  to  comment  on 
tactical  proficiency,  reinforce  good  CRM  behaviors,  and  critique  on  poor  CRM  behaviors. 

A  fifth  area  to  consider  is  MP  training.  Our  data  demonstrate  that  comprehensive  MP  by 
the  crew  is  an  extremely  important  factor  to  ensure  mission  operations  success,  but  this  activity 
is  not  being  supported  at  the  ART  level.  Specifically,  the  degree  of  importance  placed  on  MP 
correlates  to  direct  and  immediate  impact  on  mission  execution  performance.  There  is  a 
tendency,  however,  to  reduce  the  MP  period  to  a  somewhat  lesser  role  in  the  overall  training 
process.  In  this  vein,  MP  is  being  viewed  as  a  separate  training  period  fi-om  the  actual  simulator 
training  period. 

We  recommend  better  incorporation  of  advanced  planning  materials  and  semiautomated 
systems  (e.g.,  SOFPARS)  into  the  MP  process.  We  advocate  development  of  methods  for 
students  to  structure  their  own  “team”  coordination  activities  to  promote  effective  planning 
activities.  Students  need  to  be  informed  of  prescriptive  approaches  to  promote  their  effective 
mission  team  coordination  skill,  rather  than  being  allowed  to  operate  under  a  presumption  that 
good  performance  will  “just  happen”  as  they  conduct  their  normal  MP  activities.  We  would 
include  an  academic  review  session  in  ART  that  covers  effective  MP  procedures  and  techniques, 
as  well  as  methods  for  incorporation  of  advanced  planning  methods  and  materials.  Finally,  we 
would  extend  the  T^RM  concepts  into  the  MP  phase,  where  there  is  ample  time  and  opportunity 
for  instructors  to  identify  when  and  where  problem  behaviors  are  being  exhibited. 
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GLOSSARY  OF  ACRONYMS  AND  ABBREVIATIONS 


58  TRSS 

58th  Training  Support  Squadron 

ABCCC 

Airborne  Command,  Control,  and  Communications 

AC 

Aircraft  Commander 

ACT 

Aircrew  Coordination  Training 

AD 

Airdrop 

AFSOC 

Air  Force  Special  Operations  Command 

AFSOCR 

Air  Force  Special  Operations  Command  Regulation 

AFRL/HEA 

Air  Force  Research  Laboratory/  Warfighter  Training  Research  Division 

AO 

Area  of  Operation 

AOC 

Air  Operations  Center 

AR 

Air  Refueling 

ART 

Annual  Refresher  Training 

ATC 

Air  Traffic  Control 

ATO 

Air  Tasking  Order 

BARS 

Behaviorally  Anchored  Rating  Scale 

BQ 

Basic  Quality 

C3 

Command,  Control,  and  Communications 

C&C 

Command  and  Control 

CARP 

Computer  Air  Release  Point 

CAS 

Close  Air  Support 

CEOI 

Communications  Execution  Operating  Instruction 

CMAQ 

Crewmember  Management  Attitudes  Questionnaire 

CBS 

Crewmember  Background  Survey 

CMT 

Combat  Mission  Training 

COA 

Course  of  Action 

CP 

Copilot 

CRM 

Crew  Resource  Management 

CSO 

Communication  System  Operator 

DIS 

Distributed  Interactive  Simulation 

DMT 

Distributed  Mission  Training 

DZ 

Drop  Zone 

E&E 

Escape  and  Evasion 

EPs 

Emergency  Procedures 

EW 

Electronic  Warfare 

FA 

Fimction  Allocation 

FE 

Flight  Engineer 

FRAG 

Fragmentary  Order 

82 


GTM 

Groimd  Track  Map 

HTI 

Hughes  Training,  Inc. 

IDS 

Infared  Detection  System 

I/E 

Infil/Exfil 

IMOMS 

Integrated  Mini  on  Mini  System 

INS 

Inertial  Navigation  System 

INTEL 

Intelligence 

lOS 

Instructor  Operator  Station 

IP 

Instructor  Pilot 

IRI 

Instructor  Rating  Instrument 

KAFB 

Kirtland  Air  Force  Base 

LL 

Low-Level 

LM 

Loadmaster 

LN 

Left  Navigator 

LOS 

Line  Operational  Simulation 

LZ 

Landing  Zone 

MEDEVAC 

Medical  Evacuation 

MICARS 

Mission  Information  Collection,  Analysis,  and  Reporting  System 

MM-MM 

Multimeasure  Multimethod 

MOST 

Mission-Oriented  Simulator  Training 

MP 

Mission  Preparation 

MQ 

Mission  Qualification 

MR 

Mission  Rehearsal 

MRA 

Multiple  Regression  Analysis 

MTSS 

Mission  Training  Support  System 

NGAT 

Night  Guidance,  Apportionment,  and  Targeting 

NVG 

Night  Vision  Goggles 

OC 

Observer/Controller 

PF 

Pilot  Flying 

PJ 

Pararescue  Jumper 

P/0 

Participant/Observer 

R«&D 

Research  and  Development 

RN 

Right  Navigator 
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SA 

Situation  Awareness 

SAFE 

Selected  Area  for  Evasion 

SCA 

Self-Contained  Approach 

SF 

Special  Forces 

SITREP 

Situation  Report 

SME 

Subject-Matter  Expert 

SOF 

Special  Operations  Forces 

SOFI 

Special  Operations  Forces  Improvements 

SOFPARS 

Special  Operations  Forces  Preparation  and  Rehearsal  System 

SOS 

Special  Operations  Squadron 

SNS 

Satellite  Navigation  System 

STAN/EVAL 

Standardization/Evaluation 

TARGETS 

Targeted  Acceptable  Responses  to  Generated  Events  and  Tasks 

T"RM 

Tactical  Team  Resource  Management 

TE 

Tactics  Employment 

TM 

Time  Management 

TMAQl 

pre-Team  Mission  Attitudes  Questionnaire 

TMAQ2 

post-Team  Mission  Attitudes  Questionnaire 

TOLD 

Takeoff  and  Landing  Data 

TOT 

Time  on  Target 

T-MOT 

Team-Mission  Observation  Tool 

T-MPT 

Team-Mission  Performance  Tool 

T-MRAT 

Team-Mission  Readiness  Assessment  Tool 

USAF 

United  States  Air  Force 

WST 

Weapon  Systems  Trainer 
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APPENDIX  A 


Age 

MC-130P  Flying  Hrs 
Total  Flying  Hrs 
CRM  Hrs 

Hrs  Experience  in  AO 


POSITIONS 


AC 

CP 

NAV 

FE 

CSO 


Std  Deviation 


7 

1306 

1515 

42 

223 


RANKS 


Maximum 


43 

3500 

5700 

125 

500 


SQUADRON 


Age 

MC-130P  Flying  Hrs 
Total  Flying  Hrs 
CRM  Hrs 

Hrs  Experience  in  AO 


POSITIONS 


AC 

CP 

NAV 

FE 

CSO 


Minimum 


28 

200 

1500 

11 

12 


Maximum 


36 

2600 

5800 

100 

200 


SQUADRON 


CREW  #4 


CREW  #5 


CREW  #6 


86 


CREW  #7 


1 

N 

Mean 

Std  Deviation 

Minimum 

Maximum 

Age 

6 

33 

4 

28 

39 

MC-130P  Flying  Hrs 

6 

1162 

439 

500 

1600 

Total  Flying  Hrs 

6 

2262 

1036 

971 

3500 

CRMHrs 

6 

92 

82 

8 

200 

1  Hrs  Experience  in  AO 

5 

336 

707 

0 

1600 

1  POSITIONS 

RANKS 

SQUADRON  1 

AC 

1 

CP 

1 

Capt 

4 

NAV 

2 

TSgt 

1 

“B” 

SOS 

6 

FE 

1 

SSgt 

1 

CSO 

1 

CREW  #9 


N 

Mean 

Std  Deviation 

Minimum 

Maximum 

Age 

6 

35 

6 

29 

45 

MC-130P  Flying  Hrs 

6 

1375 

1257 

200 

2900 

Total  Flying  Hrs 

6 

2767 

1473 

500 

4800 

CRM  Hrs 

6 

20 

10 

30 

Hrs  Experience  in  AO 

6 

50 

99 

0 

250 

POSITIONS 

RANKS 

SQ 

UADRON 

AC 

CP 

NAV 

FE 

CSO 


1 

1 

2 

1 

1 


Ltc 

Capt 

TSgt 

SSgt 


Other 
“C”  SOS 


CREW  #10 


N 


Mean 


Std  Deviation 


Minimum 


Maximum 


Age 

MC-130P  Flying  Mrs 
Total  Flying  Hrs 
CRMHrs 

Hrs  Experience  in  AO 


6 

6 

6 

6 

6 


32 

1392 

3225 

52 

534 


3 

548 

1425 

18 

1003 


30 

700 

2300 

26 

0 


38 

1950 

6100 

75 

2500 


POSITIONS 


RANKS 


SQUADRON 


AC 

CP 

NAV 

FE 

CSO 


1 

1 

2 

1 

1 


Maj 

Capt 

SSgt 


1 

3 

2 


‘B”  SOS 


87 


CREW  #11 


CREW  #12 


CREW  #13 
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APPENDIX  B 


Examples  of  TE,  FA,  SA,  and  C3  Subprocess  Assessments  from  the  T-MOT. 


Tactics  Employment  (TE):  Includes  all  analytic  activities  necessary  to  avoid  or  minimize  threat 
detection,  or  exposure,  and  to  successfully  coordinate  complex  mission  events  and  multipie  mission  ob¬ 
jectives. 


2.0  There  are  (typically)  three  tactical  options  to  use  in  order  to  go  undetected:  Altitude,  Airspeed,  and  Terrain. 


a.  Was  a  particular  mix  of  tactics  options  considered? .  YES  /  NO 

(Explain)  _ _ _ _ _ 

b.  Did  the  crew  change  the  tactics  options  as  a  function  of  difficulty  in  each  mission  phase? .  YES/NO 

(Explain) _ _ 

c.  Was  one  option  (e.g.,  speed)  preferred  over  the  others? .  YES/NO 

(Explain) _ _ _ 


d.  Did  any  crewmember  periodically  review  or  verify  the  status  of  the  threat  planning  strategy? . YES/NO 

(Explain) _ _ _ 


Function  Allocation  (FA):  Includes  the  division  of  crew  responsibilities  so  that  workload  is 
distributed  among  the  crew,  avoiding  redundant  tasking,  task  overload,  and  crewmember  disinterest  or 
noninvolvement.  Tasks  should  be  allocated  in  such  a  manner  so  that  crewmembers  are  able  to  share 
information  and  coordinate  responsibilities. 


3.0 


Workload  and/or  task  distribution  should  be  clearly  communicated  and 
crewmembers. 

a.  Was  the  mission  workload  distribution  clearly  communicated  and  acknowledged? . 

(Explain)  _ _ ” 

b.  Were  secondary  tasks  prioritized  so  as  to  allow  sufficient  resources  for  primary  tasks?,.. 

(Explain) _ 

c.  Did  nonoperational  factors  (such  as  social  interaction)  interfere  with  any  crewmember  Is 

abilities  while  performing  necessary  tasks? . 

(Explain _ _ 


acknowledged 
.  YES/NO 

IZ7  YES/NO 
YES/NO 


by 


Situation  Awareness  (SA):  Entails  maintaining  an  accurate  mental  picture  of  mission  events  and 
objectives  as  they  unfold  over  time  and  space.  Emphasis  and  analysis  are  placed  on  the  three  levels  of 
SA  (perception,  integration,  and  generation:  Endsley,  1995b)  and  their  impact  on  team  coordination. 


4.0  At  least  one  crewmember’s  overall  SA  should  be  high,  and  an  assessment  of  mission  difficulty 
should  be  made  based  on  (for  example):  marginal  weather,  threat  saturation  is  high,  large  no. 
of  mission  events,  etc. 


a.  Did  any  individual  crewmember  indicate  an  overall  assessment  of  mission  difficulty? .  YES/NO 

(Explain) _ _ _ _ _ 

b.  Did  crewmember(s)  prepare  for  unexpected  or  contingency  situations? .  YES/NO 

(Explain) _ 


Command,  Control,  and  Communications  (C3):  Encompasses  those  activities  required  to  involve 
external  parties  in  the  mission,  and  to  maintain  communications  with  these  external  team  members; 
communication  within  the  crew;  and  controlling  the  sequence  of  mission  events  according  to  the  missiori 
execution  plan. 
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5.0  Crew's  willingness  to  challenge  the  system. 

a.  Do  crewmembers  request  specific  resources  they  need? .  YES/NO 

(Explain) _ 

b.  Do  crewmembers  question/challenge  assumptions  (e.g.,  within  frag,  threat  SITREP,  etc.)? .  YES/NO 

(Explain) _ _ _ 

c.  Do  crewmembers  ferret  out  needed  materials  and  information  from  all  sources? .  YES/NO 

(Explain) _ 
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